Goal based educational system utilizing a remediation object

ABSTRACT

A system is disclosed that provides a goal based learning system utilizing a rule based expert training system to provide a cognitive educational experience. The system provides the user with a simulated environment that presents a business opportunity to understand and solve optimally. Mistakes are noted and remedial educational material presented dynamically to build the necessary skills that a user requires for success in the business endeavor. The system utilizes an artificial intelligence engine driving individualized and dynamic feedback with synchronized video and graphics used to simulate real-world environment and interactions. Multiple “correct” answers are integrated into the learning system to allow individualized learning experiences in which navigation through the system is at a pace controlled by the learner. A robust business model provides support for realistic activities and allows a user to experience real world consequences for their actions and decisions and a tutorial system analyzes student inputs to determine appropriate feedback to teach new skills.

This is a Continuation application of prior application Ser. No.09/218,741 filed on Dec. 22. 1998, U.S. Pat. No. 6,029,158, whichdesignated the United States, the disclosure of which is incorporatedherein by reference.

FIELD OF THE INVENTION

The present invention relates to education systems and more particularlyto a tutorial system that analyzes student inputs to determineappropriate feedback to teach new skills.

BACKGROUND OF THE INVENTION

When building a knowledge based system or expert system, at least twodisciplines are necessary to properly construct the rules that drive theknowledge base, the discipline of the knowledge engineer and theknowledge of the expert. The domain expert has knowledge of the domainor field of use of the expert system. For example, the domain expert ofan expert for instructing students in an automotive manufacturingfacility might be a process control engineer while the domain expert fora medical instruction system might be a doctor or a nurse. The knowledgeengineer is a person that understands the expert system and utilizes theexpert's knowledge to create an application for the system. In manyinstances, the knowledge engineer and domain expert are separate peoplewho have to collaborate to construct the expert system.

Typically, this collaboration takes the form of the knowledge engineerasking questions of the domain expert and incorporating the answers tothese questions into the design of the system. This approach is laborintensive, slow and error prone. The coordination of the two separatedisciplines may lead to problems. Although the knowledge engineer cantranscribe input from the expert utilizing videotape, audio tape, textand other sources, efforts from people of both disciplines have to beexpended. Further, if the knowledge engineer does not ask the rightquestions or asks the questions in an incorrect way, the informationutilized to design the knowledge base could be incorrect. Feedback tothe knowledge engineer from the expert system is often not available inprior art system until the construction is completed. With conventionalsystem, there is a time consuming feedback loop that ties togethervarious processes from knowledge acquisition to validation.

Educational systems utilizing an expert system component often sufferfrom a lack of motivational aspects that result in a user becoming boredor ceasing to complete a training program. Current training programsutilize static, hard-coded feedback with some linear video and graphicsused to add visual appeal and illustrate concepts. These systemstypically support one “correct” answer and navigation through the systemis only supported through a single defined path which results in atwo-dimensional generic interaction, with no business model support anda single feedback to the learner of correct or incorrect based on theselected response. Current tutorial systems do not architect realbusiness simulations into the rules to provide a creative learningenvironment to a user.

SUMMARY OF THE INVENTION

According to a broad aspect of a preferred embodiment of the invention,a goal based learning system utilizes a rule based expert trainingsystem to provide a cognitive educational experience. The systemprovides the user with a simulated environment that presents a businessopportunity to understand and solve optimally. Mistakes are noted andremedial educational material presented dynamically to build thenecessary skills that a user requires for success in the businessendeavor. The system utilizes an artificial intelligence engine drivingindividualized and dynamic feedback with synchronized video and graphicsused to simulate real-world environment and interactions. Multiple“correct” answers are integrated into the learning system to allowindividualized learning experiences in which navigation through thesystem is at a pace controlled by the learner. A robust business modelprovides support for realistic activities and allows a user toexperience real world consequences for their actions and decisions and atutorial system analyzes student inputs to determine appropriatefeedback to teach new skills.

DESCRIPTION OF THE DRAWINGS

The foregoing and other objects, aspects and advantages are betterunderstood from the following detailed description of a preferredembodiment of the invention with reference to the drawings, in which:

FIG. 1 is a block diagram of a representative hardware environment inaccordance with a preferred embodiment;

FIG. 2 is a block diagram of a system architecture in accordance with apreferred embodiment;

FIG. 3 depicts the timeline and relative resource requirements for eachphase of development for a typical application development in accordancewith a preferred embodiment;

FIG. 4 depicts the potential savings in both functional and technicaltasks in accordance with a preferred embodiment;

FIG. 5 illustrates commonalties in accordance with a preferredembodiment;

FIG. 6 illustrates a development architecture approach in accordancewith a preferred embodiment;

FIG. 7 illustrates a small segment of a domain model for claims handlersin the auto insurance industry in accordance with a preferredembodiment;

FIG. 8 illustrates an instantiated domain model in accordance with apreferred embodiment;

FIG. 9 illustrates an insurance underwriting profile in accordance witha preferred embodiment;

FIG. 10 illustrates a transformation component in accordance with apreferred embodiment;

FIG. 11 illustrates the use of a toolbar to navigate and accessapplication level features in accordance with a preferred embodiment;

FIG. 12 is a GBS display in accordance with a preferred embodiment;

FIG. 13 is a feedback display in accordance with a preferred embodiment;

FIG. 14 illustrates a display in which a student has made some mistakesin accordance with a preferred embodiment;

FIG. 15 illustrates a journal entry simulation in accordance with apreferred embodiment;

FIG. 16 illustrates a simulated Bell Phone Bill journal entry inaccordance with a preferred embodiment;

FIG. 17 illustrates a feedback display in accordance with a preferredembodiment;

FIGS. 18 and 19 illustrate a feedback display in accordance with apreferred embodiment;

FIG. 20 illustrates a feedback display in accordance with a preferredembodiment;

FIG. 21 illustrates a simulation display in accordance with a preferredembodiment;

FIG. 22 illustrates the steps of the first scenario in accordance with apreferred embodiment;

FIGS. 23 and 24 illustrate the steps associated with a build scenario inaccordance with a preferred embodiment;

FIG. 25 illustrates how the tool suite supports student administrationin accordance with a preferred embodiment;

FIG. 26 illustrates a suite to support a student interaction inaccordance with a preferred embodiment;

FIG. 27 illustrates the remediation process in accordance with apreferred embodiment;

FIG. 28 illustrates a display of journalization transactions inaccordance with a preferred embodiment;

FIG. 29 illustrates the objects for the journalization task inaccordance with a preferred embodiment;

FIG. 30 illustrates the mapping of a source item to a target item inaccordance with a preferred embodiment;

FIG. 31 illustrates target group bundles in accordance with a preferredembodiment;

FIG. 32 illustrates a TargetGroup Hierarchy in accordance with apreferred embodiment;

FIG. 33 illustrates a small section the amount of feedback in accordancewith a preferred embodiment;

FIG. 34 illustrates an analysis of rules in accordance with a preferredembodiment;

FIG. 35 illustrates a feedback selection in accordance with a preferredembodiment;

FIG. 36 is a flowchart of the feedback logic in accordance with apreferred embodiment;

FIG. 37 illustrates an example of separating out some mistakes for theinterface to catch and others for the ICAT to catch has positive andnegative impacts in accordance with a preferred embodiment;

FIG. 38 is a block diagram of the hierarchical relationship of atransaction in accordance with a preferred embodiment;

FIG. 39 is a block diagram illustrating the feedback hierarchy inaccordance with a preferred embodiment;

FIG. 40 is a block diagram illustrating how the simulation engine isarchitected into a preferred embodiment of the invention;

FIG. 41 is a block diagram setting forth the architecture of asimulation model in accordance with a preferred embodiment;

FIG. 42 illustrates the arithmetic steps in accordance with a preferredembodiment;

FIG. 43 illustrates a drag & drop input operation in accordance with apreferred embodiment;

FIG. 44 illustrates list object processing in accordance with apreferred embodiment;

FIG. 45 illustrates the steps for configuring a simulation in accordancewith a preferred embodiment;

FIG. 46 illustrates a distinct output in accordance with a preferredembodiment;

FIG. 47 is a block diagram presenting the detailed architecture of asystem dynamics model in accordance with a preferred embodiment;

FIG. 48 is graphical representation of the object model which isutilized to instantiate the system dynamic engine in accordance with apreferred embodiment.

FIG. 49 is a PInput Cell for a simulation model in accordance with apreferred embodiment;

FIG. 50 is a PInput backup cell in a simulation model in accordance witha preferred embodiment;

FIG. 51 is a display illustrating a POutput cell in accordance with apreferred embodiment. The steps required to configure the POutput arepresented below;

FIG. 52 is an overview diagram of the logic utilized for initialconfiguration in accordance with a preferred embodiment;

FIG. 53 is a display of the source item and target configuration inaccordance with a preferred embodiment;

FIG. 54 is a display of video information in accordance with a preferredembodiment;

FIG. 55 illustrates a display depicting configured rules in accordancewith a preferred embodiment;

FIG. 56 illustrates feedback for configured rules in accordance with apreferred embodiment;

FIG. 57 illustrates a display with follow-up configuration questions inaccordance with a preferred embodiment;

FIG. 58 illustrates configuration of aggregate rules in accordance witha preferred embodiment;

FIG. 59 illustrates a set of coach items in accordance with a preferredembodiment;

FIG. 60 is an ICA Meeting Configuration tool display in accordance witha preferred embodiment;

FIG. 61 illustrates an ICA utility in accordance with a preferredembodiment;

FIG. 62 illustrates a configuration utility display in accordance with apreferred embodiment;

FIG. 63 illustrates an object editor toolbar in accordance with apreferred embodiment;

FIG. 64 illustrates the seven areas that can be configured for asimulation in accordance with a preferred embodiment;

FIG. 65 illustrates a display that defines inputs in accordance with apreferred embodiment;

FIG. 66 illustrates a list editor in accordance with a preferredembodiment;

FIG. 67A illustrates a define student display in accordance with apreferred embodiment;

FIG. 67B illustrates a ControlSourceItem display in accordance with apreferred embodiment;

FIG. 68 illustrates a simulation workbench in accordance with apreferred embodiment;

FIG. 69 illustrates an object viewer in accordance with a preferredembodiment. As shown in

FIG. 70 illustrates an Object Viewer Configuration in an Utilities menuin accordance with a preferred embodiment;

FIG. 71 illustrates a log viewer in accordance with a preferredembodiment;

FIG. 72 illustrates a Doc Maker display in accordance with a preferredembodiment;

FIG. 73 illustrates a Feedback display in accordance with a preferredembodiment;

FIG. 74 is an object editor display that illustrates the use ofreferences in accordance with a preferred embodiment; and

FIG. 75 presents the detailed design of smart spreadsheets in accordancewith a preferred embodiment.

DETAILED DESCRIPTION

A preferred embodiment of a system in accordance with the presentinvention is preferably practiced in the context of a personal computersuch as an IBM compatible personal computer, Apple Macintosh computer orUNIX based workstation. A representative hardware environment isdepicted in FIG. 1, which illustrates a typical hardware configurationof a workstation in accordance with a preferred embodiment having acentral processing unit 110, such as a microprocessor, and a number ofother units interconnected via a system bus 112. The workstation shownin FIG. 1 includes a Random Access Memory (RAM) 114, Read Only Memory(ROM) 116, an I/O adapter 118 for connecting peripheral devices such asdisk storage units 120 to the bus 112, a user interface adapter 122 forconnecting a keyboard 124, a mouse 126, a speaker 128, a microphone 132,and/or other user interface devices such as a touch screen (not shown)to the bus 112, communication adapter 134 for connecting the workstationto a communication network (e.g., a data processing network) and adisplay adapter 136 for connecting the bus 112 to a display device 138.The workstation typically has resident thereon an operating system suchas the Microsoft Windows NT or Windows/95 Operating System (OS), the IBMOS/2 operating system, the MAC OS, or UNIX operating system. Thoseskilled in the art will appreciate that the present invention may alsobe implemented on platforms and operating systems other than thosementioned.

A preferred embodiment is written using JAVA, C, and the C++ languageand utilizes object oriented programming methodology. Object orientedprogramming (OOP) has become increasingly used to develop complexapplications. As OOP moves toward the mainstream of software design anddevelopment, various software solutions require adaptation to make useof the benefits of OOP. A need exists for these principles of OOP to beapplied to a messaging interface of an electronic messaging system suchthat a set of OOP classes and objects for the messaging interface can beprovided.

OOP is a process of developing computer software using objects,including the steps of analyzing the problem, designing the system, andconstructing the program. An object is a software package that containsboth data and a collection of related structures and procedures. Sinceit contains both data and a collection of structures and procedures, itcan be visualized as a self-sufficient component that does not requireother additional structures, procedures or data to perform its specifictask. OOP, therefore, views a computer program as a collection oflargely autonomous components, called objects, each of which isresponsible for a specific task. This concept of packaging data,structures, and procedures together in one component or module is calledencapsulation.

In general, OOP components are reusable software modules which presentan interface that conforms to an object model and which are accessed atrun-time through a component integration architecture. A componentintegration architecture is a set of architecture mechanisms which allowsoftware modules in different process spaces to utilize each otherscapabilities or functions. This is generally done by assuming a commoncomponent object model on which to build the architecture. It isworthwhile to differentiate between an object and a class of objects atthis point. An object is a single instance of the class of objects,which is often just called a class. A class of objects can be viewed asa blueprint, from which many objects can be formed.

OOP allows the programmer to create an object that is a part of anotherobject. For example, the object representing a piston engine is said tohave a composition-relationship with the object representing a piston.In reality, a piston engine comprises a piston, valves and many othercomponents; the fact that a piston is an element of a piston engine canbe logically and semantically represented in OOP by two objects.

OOP also allows creation of an object that “depends from” anotherobject. If there are two objects, one representing a piston engine andthe other representing a piston engine wherein the piston is made ofceramic, then the relationship between the two objects is not that ofcomposition. A ceramic piston engine does not make up a piston engine.Rather it is merely one kind of piston engine that has one morelimitation than the piston engine; its piston is made of ceramic. Inthis case, the object representing the ceramic piston engine is called aderived object, and it inherits all of the aspects of the objectrepresenting the piston engine and adds further limitation or detail toit. The object representing the ceramic piston engine “depends from” theobject representing the piston engine. The relationship between theseobjects is called inheritance.

When the object or class representing the ceramic piston engine inheritsall of the aspects of the objects representing the piston engine, itinherits the thermal characteristics of a standard piston defined in thepiston engine class. However, the ceramic piston engine object overridesthese ceramic specific thermal characteristics, which are typicallydifferent from those associated with a metal piston. It skips over theoriginal and uses new functions related to ceramic pistons. Differentkinds of piston engines have different characteristics, but may have thesame underlying functions associated with it (e.g., how many pistons inthe engine, ignition sequences, lubrication, etc.). To access each ofthese functions in any piston engine object, a programmer would call thesame functions with the same names, but each type of piston engine mayhave different/overriding implementations of functions behind the samename. This ability to hide different implementations of a functionbehind the same name is called polymorphism and it greatly simplifiescommunication among objects.

With the concepts of composition-relationship, encapsulation,inheritance and polymorphism, an object can represent just aboutanything in the real world. In fact, our logical perception of thereality is the only limit on determining the kinds of things that canbecome objects in object-oriented software. Some typical categories areas follows:

Objects can represent physical objects, such as automobiles in atraffic-flow simulation, electrical components in a circuit-designprogram, countries in an economics model, or aircraft in anair-traffic-control system.

Objects can represent elements of the computer-user environment such aswindows, menus or graphics objects.

An object can represent an inventory, such as a personnel file or atable of the latitudes and longitudes of cities.

An object can represent user-defined data types such as time, angles,and complex numbers, or points on the plane.

With this enormous capability of an object to represent just about anylogically separable matters, OOP allows the software developer to designand implement a computer program that is a model of some aspects ofreality, whether that reality is a physical entity, a process, a system,or a composition of matter. Since the object can represent anything, thesoftware developer can create an object which can be used as a componentin a larger software project in the future.

If 90% of a new OOP software program consists of proven, existingcomponents made from preexisting reusable objects, then only theremaining 10% of the new software project has to be written and testedfrom scratch. Since 90% already came from an inventory of extensivelytested reusable objects, the potential domain from which an error couldoriginate is 10% of the program. As a result, OOP enables softwaredevelopers to build objects out of other, previously built objects.

This process closely resembles complex machinery being built out ofassemblies and sub-assemblies. OOP technology, therefore, makes softwareengineering more like hardware engineering in that software is builtfrom existing components, which are available to the developer asobjects. All this adds up to an improved quality of the software as wellas an increased speed of its development.

Programming languages are beginning to fully support the OOP principles,such as encapsulation, inheritance, polymorphism, andcomposition-relationship. With the advent of the C++ language, manycommercial software developers have embraced OOP. C++ is an OOP languagethat offers a fast, machine-executable code. Furthermore, C++ issuitable for both commercial-application and systems-programmingprojects. For now, C++ appears to be the most popular choice among manyOOP programmers, but there is a host of other OOP languages, such asSmalltalk, Common Lisp Object System (CLOS), and Eiffel. Additionally,OOP capabilities are being added to more traditional popular computerprogramming languages such as Pascal.

The benefits of object classes can be summarized, as follows:

Objects and their corresponding classes break down complex programmingproblems into many smaller, simpler problems.

Encapsulation enforces data abstraction through the organization of datainto small, independent objects that can communicate with each other.Encapsulation protects the data in an object from accidental damage, butallows other objects to interact with that data by calling the object'smember functions and structures.

Subclassing and inheritance make it possible to extend and modifyobjects through deriving new kinds of objects from the standard classesavailable in the system. Thus, new capabilities are created withouthaving to start from scratch.

Polymorphism and multiple inheritance make it possible for differentprogrammers to mix and match characteristics of many different classesand create specialized objects that can still work with related objectsin predictable ways.

Class hierarchies and containment hierarchies provide a flexiblemechanism for modeling real-world objects and the relationships amongthem.

Libraries of reusable classes are useful in many situations, but theyalso have some limitations. For example:

Complexity. In a complex system, the class hierarchies for relatedclasses can become extremely confusing, with many dozens or evenhundreds of classes.

Flow of control. A program written with the aid of class libraries isstill responsible for the flow of control (i.e., it must control theinteractions among all the objects created from a particular library).The programmer has to decide which functions to call at what times forwhich kinds of objects.

Duplication of effort. Although class libraries allow programmers to useand reuse many small pieces of code, each programmer puts those piecestogether in a different way. Two different programmers can use the sameset of class libraries to write two programs that do exactly the samething but whose internal structure (i.e., design) may be quitedifferent, depending on hundreds of small decisions each programmermakes along the way. Inevitably, similar pieces of code end up doingsimilar things in slightly different ways and do not work as welltogether as they should.

Class libraries are very flexible. As programs grow more complex, moreprogrammers are forced to reinvent basic solutions to basic problemsover and over again. A relatively new extension of the class libraryconcept is to have a framework of class libraries. This framework ismore complex and consists of significant collections of collaboratingclasses that capture both the small scale patterns and major mechanismsthat implement the common requirements and design in a specificapplication domain. They were first developed to free applicationprogrammers from the chores involved in displaying menus, windows,dialog boxes, and other standard user interface elements for personalcomputers.

Frameworks also represent a change in the way programmers think aboutthe interaction between the code they write and code written by others.In the early days of procedural programing, the programmer calledlibraries provided by the operating system to perform certain tasks, butbasically the program executed down the page from start to finish, andthe programmer was solely responsible for the flow of control. This wasappropriate for printing out paychecks, calculating a mathematicaltable, or solving other problems with a program that executed in justone way.

The development of graphical user interfaces began to turn thisprocedural programming arrangement inside out. These interfaces allowthe user, rather than program logic, to drive the program and decidewhen certain actions should be performed. Today, most personal computersoftware accomplishes this by means of an event loop which monitors themouse, keyboard, and other sources of external events and calls theappropriate parts of the programmer's code according to actions that theuser performs. The programmer no longer determines the order in whichevents occur. Instead, a program is divided into separate pieces thatare called at unpredictable times and in an unpredictable order. Byrelinquishing control in this way to users, the developer creates aprogram that is much easier to use. Nevertheless, individual pieces ofthe program written by the developer still call libraries provided bythe operating system to accomplish certain tasks, and the programmermust still determine the flow of control within each piece after it'scalled by the event loop. Application code still “sits on top of” thesystem.

Even event loop programs require programmers to write a lot of code thatshould not need to be written separately for every application. Theconcept of an application framework carries the event loop conceptfurther. Instead of dealing with all the nuts and bolts of constructingbasic menus, windows, and dialog boxes and then making these things allwork together, programmers using application frameworks start withworking application code and basic user interface elements in place.Subsequently, they build from there by replacing some of the genericcapabilities of the framework with the specific capabilities of theintended application.

Application frameworks reduce the total amount of code that a programmerhas to write from scratch. However, because the framework is really ageneric application that displays windows, supports copy and paste, andso on, the programmer can also relinquish control to a greater degreethan event loop programs permit. The framework code takes care of almostall event handling and flow of control, and the programmer's code iscalled only when the framework needs it (e.g., to create or manipulate aproprietary data structure).

A programmer writing a framework program not only relinquishes controlto the user (as is also true for event loop programs), but alsorelinquishes the detailed flow of control within the program to theframework. This approach allows the creation of more complex systemsthat work together in interesting ways, as opposed to isolated programs,having custom code, being created over and over again for similarproblems.

Thus, as is explained above, a framework basically is a collection ofcooperating classes that make up a reusable design solution for a givenproblem domain. It typically includes objects that provide defaultbehavior (e.g., for menus and windows), and programmers use it byinheriting some of that default behavior and overriding other behaviorso that the framework calls application code at the appropriate times.

There are three main differences between frameworks and class libraries:

Behavior versus protocol. Class libraries are essentially collections ofbehaviors that you can call when you want those individual behaviors inyour program. A framework, on the other hand, provides not only behaviorbut also the protocol or set of rules that govern the ways in whichbehaviors can be combined, including rules for what a programmer issupposed to provide versus what the framework provides.

Call versus override. With a class library, the code the programmerinstantiates objects and calls their member functions. It's possible toinstantiate and call objects in the same way with a framework (i.e., totreat the framework as a class library), but to take full advantage of aframework's reusable design, a programmer typically writes code thatoverrides and is called by the framework. The framework manages the flowof control among its objects. Writing a program involves dividingresponsibilities among the various pieces of software that are called bythe framework rather than specifying how the different pieces shouldwork together.

Implementation versus design. With class libraries, programmers reuseonly implementations, whereas with frameworks, they reuse design. Aframework embodies the way a family of related programs or pieces ofsoftware work. It represents a generic design solution that can beadapted to a variety of specific problems in a given domain. Forexample, a single framework can embody the way a user interface works,even though two different user interfaces created with the sameframework might solve quite different interface problems.

Thus, through the development of frameworks for solutions to variousproblems and programming tasks, significant reductions in the design anddevelopment effort for software can be achieved. A preferred embodimentof the invention utilizes HyperText Markup Language (HTML) to implementdocuments on the Internet together with a general-purpose securecommunication protocol for a transport medium between the client and theNewco. HTTP or other protocols could be readily substituted for HTMLwithout undue experimentation. Information on these products isavailable in T. Berners-Lee, D. Connoly, “RFC 1866: Hypertext MarkupLanguage-2.0” (November 1995); and R. Fielding, H, Frystyk, T.Berners-Lee, J. Gettys and J. C. Mogul, “Hypertext TransferProtocol—HTTP/1.1:HTTP Working Group Internet Draft” (May 2, 1996). HTMLis a simple data format used to create hypertext documents that areportable from one platform to another. HTML documents are SGML documentswith generic semantics that are appropriate for representing informationfrom a wide range of domains. HTML has been in use by the World-Wide Webglobal information initiative since 1990. HTML is an application of ISOStandard 8879; 1986 Information Processing Text and Office Systems;Standard Generalized Markup Language (SGML).

To date, Web development tools have been limited in their ability tocreate dynamic Web applications which span from client to server andinteroperate with existing computing resources. Until recently, HTML hasbeen the dominant technology used in development of Web-based solutions.However, HTML has proven to be inadequate in the following areas:

Poor performance;

Restricted user interface capabilities;

Can only produce static Web pages;

Lack of interoperability with existing applications and data; and

Inability to scale.

Sun Microsystem's Java language solves many of the client-side problemsby:

Improving performance on the client side;

Enabling the creation of dynamic, real-time Web applications; and

Providing the ability to create a wide variety of user interfacecomponents.

With Java, developers can create robust User Interface (UI) components.Custom “widgets” (e.g., real-time stock tickers, animated icons, etc.)can be created, and client-side performance is improved. Unlike HTML,Java supports the notion of client-side validation, offloadingappropriate processing onto the client for improved performance.Dynamic, real-time Web pages can be created. Using the above-mentionedcustom UI components, dynamic Web pages can also be created.

Sun's Java language has emerged as an industry-recognized language for“programming the Internet.” Sun defines Java as: “a simple,object-oriented, distributed, interpreted, robust, secure,architecture-neutral, portable, high-performance, multithreaded,dynamic, buzzword-compliant, general-purpose programming language. Javasupports programming for the Internet in the form ofplatform-independent Java applets.” Java applets are small, specializedapplications that comply with Sun's Java Application ProgrammingInterface (API) allowing developers to add “interactive content” to Webdocuments (e.g., simple animations, page adornments, basic games, etc.).Applets execute within a Java-compatible browser (e.g., NetscapeNavigator) by copying code from the server to client. From a languagestandpoint, Java's core feature set is based on C++. Sun's Javaliterature states that Java is basically, “C++ with extensions fromObjective C for more dynamic method resolution.”

Another technology that provides similar function to JAVA is provided byMicrosoft and ActiveX Technologies, to give developers and Web designerswherewithal to build dynamic content for the Internet and personalcomputers. ActiveX includes tools for developing animation, 3-D virtualreality, video and other multimedia content. The tools use Internetstandards, work on multiple platforms, and are being supported by over100 companies. The group's building blocks are called ActiveX Controls,small, fast components that enable developers to embed parts of softwarein hypertext markup language (HTML) pages. ActiveX Controls work with avariety of programming languages including Microsoft Visual C++, BorlandDelphi, Microsoft Visual Basic programming system and, in the future,Microsoft's development tool for Java, code named “Jakarta.” ActiveXTechnologies also includes ActiveX Server Framework, allowing developersto create server applications. One of ordinary skill in the art readilyrecognizes that ActiveX could be substituted for JAVA without undueexperimentation to practice the invention.

A simulation engine in accordance with a preferred embodiment is basedon a Microsoft Visual Basic component developed to help design and testfeedback in relation to a Microsoft Excel spreadsheet. These spreadsheetmodels are what simulate actual business functions and become a taskthat will be performed by a student The Simulation Engine acceptssimulation inputs and calculates various outputs and notifies the systemof the status of the simulation at a given time in order to obtainappropriate feedback.

Relationship of Components

The simulation model executes the business function that the student islearning and is therefore the center point of the application. Anactivity ‘layer’ allows the user to visually guide the simulation bypassing inputs into the simulation engine and receiving an output fromthe simulation model. For example, if the student was working on anincome statement activity, the net sales and cost of goods soldcalculations are passed as inputs to the simulation model and the netincome value is calculated and retrieved as an output. As calculationsare passed to and retrieved from the simulation model, they are alsopassed to the Intelligent Coaching Agent (ICA). The ICA analyzes theInputs and Outputs to the simulation model and generates feedback basedon a set of rules. This feedback is received and displayed through theVisual Basic Architecture.

FIG. 2 is a block diagram of a system architecture in accordance with apreferred embodiment. The Presentation ‘layer’ 210 is separate from theactivity ‘layer’ 220 and communication is facilitated through a set ofmessages 230 that control the display specific content topics. Apreferred embodiment enables knowledge workers 200 & 201 to acquirecomplex skills rapidly, reliably and consistently across an organizationto deliver rapid acquisition of complex skills.

This result is achieved by placing individuals in a simulated businessenvironment that “looks and feels” like real work, and challenging themto make decisions which support a business' strategic objectivesutilizing highly effective learning theory (e.g., goal based learning,learn by doing, failure based learning, etc.), and the latest inmultimedia user interfaces, coupled with three powerful, integratedsoftware components. The first of these components is a softwareSolution Construction Aid (SCA) 230 consisting of a mathematicalmodeling tool 234 which simulates business outcomes of an individual'scollective actions over a period of time. The second component is aknowledge system 250 consisting of an HTML content layer which organizesand presents packaged knowledge much like an online text book withpractice exercises, video war stories, and a glossary. The thirdcomponent is a software tutor 270 comprising an artificial intelligenceengine 240 which generates individualized coaching messages based ondecisions made by learner.

Feedback is unique for each individual completing the course andsupports client cultural messages 242 “designed into” the course. Abusiness simulation methodology that includes support for contentacquisition, story line design, interaction design, feedback andcoaching delivery, and content delivery is architected into the systemin accordance with a preferred embodiment. A large number of“pre-designed” learning interactions such as drag and drop associationof information 238, situation assessment/action planning, interviewing(one-on-one, one-to-many), presenting (to a group ofexperts/executives), metering of performance candle now, handle later),“time jumping” for impact of decisions, competitive landscape shift(while “time jumping”, competitors merge, customers are acquired, etc.)and video interviewing with automated note taking are also included inaccordance with a preferred embodiment.

Business simulation in accordance with a preferred embodiment deliverstraining curricula in an optimal manner. This is because suchapplications provide effective training that mirrors a student's actualwork environment. The application of skills “on the job” facilitatesincreased retention and higher overall job performance. While theresults of such training applications are impressive, businesssimulations are very complex to design and build correctly. Thesesimulations are characterized by a very open-ended environment, wherestudents can go through the application along any number of paths,depending on their learning style and prior experiences/knowledge.

A category of learning approaches called Learn by Doing, is commonlyused as a solution to support the first phase (Learn) of the WorkforcePerformance Cycle. However, it can also be a solution to support thesecond phase (Perform) of the cycle to enable point of need learningduring job performance. By adopting the approach presented, some of thebenefits of a technology based approach for building business simulationsolutions which create more repeatable, predictable projects resultingin more perceived and actual user value at a lower cost and in less timeare highlighted.

Most corporate training programs today are misdirected because they havefailed to focus properly on the purpose of their training. Theseprograms have confused the memorization of facts with the ability toperform tasks; the knowing of “that” with the knowing of “how”. Byadopting the methods of traditional schools, businesses are teaching awide breadth of disconnected, decontextualized facts and figures, whenthey should be focused on improved performance. How do you teachperformance, when lectures, books, and tests inherently are designedaround facts and figures? Throw away the lectures, books, and tests. Thebest way to prepare for high performance is to perform; experience isthe best teacher! Most business leaders agree that workers become moreeffective the more time they spend in their jobs. The best approach fortraining novice employees, therefore, would be letting them learn on thejob, acquiring skills in their actual work environment. The idea oflearning-by-doing is not revolutionary, yet it is resisted in businessand academia. Why is this so, if higher competence is universallydesired?

Learners are reluctant to adopt learning-by-doing because they arefrightened of failure. People work hard to avoid making mistakes infront of others. Business leaders are hesitant to implementlearning-by-doing because novice failure may have dramatic safety, legaland financial implications. Imagine a novice pilot learning-by-doing ashe accelerates a large jet plane down a runway; likewise, consider a newfinancial analyst learning-by-doing as he structures a multi-milliondollar financial loan. Few employers are willing to endure such failuresto have a more competent workforce.

The concerns of employee and employer can be relieved if the training(and accompanying failure) didn't occur in front of co-workers andclients, if it didn't jeopardize a multi-million dollar aircraft or amulti-million dollar deal. What if the training was performed privately,in a richly modeled simulation of the workers actual job? In a simulatedenvironment, failure would result in dedicated instruction instead ofembarrassment, injury, or monetary losses. Simulated environmentsprovide a sense of liberation for exploration that does not exist in thereal world. Knowing that the consequences of experimentation willunlikely be dire, learners are able to explore the “what if . . . ”inherent in us all. In this way, the best way to prepare for highperformance is to simulate actual performance. New media technologiesutilizing multimedia provide the possibility to create such businesssimulation experiences.

Even if companies didn't make the mistake of focusing on “what” learningvs. “how” learning, they still tend to have the overly narrow view ofeducation/training as something that only occurs prior to workers beingasked to actually perform their job. Learning is something that isconstantly occurring, and doesn't cease once “real work” has begun. Thecloser new lessons occur in time with the application of those lessons,the greater the resultant learning. This fact helps to explain some ofthe reasoning behind the benefits of business simulation, described inthe previous section. In those systems, all new lessons are taught inclose relationship to their real world use; everything is in contextand, most importantly, are presented at the appropriate time. But as theproperly trained worker performs their job, the working environmentchanges. New demands occur, and new methods and rules are developed. Asthese events occur, there is a need for new support/training that, inmost cases, must wait to be addressed until the next “pre-performance”training session.

In these cases, the need (or demand) for additional training doesn'tmatch the supply. This lag between a training need and the fulfillinglesson has a dramatic negative impact on productivity, accuracy, andcustomer satisfaction. Workers need to have the opportunity to learnwhile they are performing. Just as during pre-performance training, onepowerful mechanism for identifying and correcting (simulated)performance problems is to have an expert available at all time to watchyour actions and remediate when appropriate. This, obviously, is toocostly and time intensive of an approach to be practical with actualexperts. But what if workers had access to a support system thatprovided the majority of the benefits of a real expert, transparentlyintegrated into their work environment? Such a system would provideadvice at key moments in the work flow for problem resolution and/orprocess improvement, tools to ease task completion, reference materialof best practice knowledge, and point of need training courses. With asupport system that proactively assists the worker in performance oftheir job tasks at a higher level of competency, productivity andcustomer satisfaction (both internal and external) would soar.

The key to such a support system is that it is seamlessly integratedinto the business system that the knowledge worker uses to execute theirjob tasks. Workers don't need to go “off-line” or seek out crypticinformation buried within paper manuals and binders for guidance or tofind the answer to queries. All the support components are madeavailable through the same applications the worker's use, at the pointin which they need them, tailored to the individual to show “how”, notjust “what”. Learning would be occurring all the time, with littledistinction between performing and improving performance.

Establishing that training should focus on performance (how), ratherthan facts (what), and extending the model of learning to includeassistance while performing, rather than only before performance, stillleaves us dangerously exposed in preparing to compete in the new,chaotic economy. As was mentioned in the opening of this paper, the paceof change in business today is whiplash fast. Not only are new methodsof doing business evolving every 18-24 months, new competitors emerge,dominate, and fade in time periods businesses used to take to performdemographic studies. Now more than ever, those who do not reinventthemselves on a regular basis will be fossilized by the pace of change.

Even the best pre-performance training and the most advanced performancesupport tools are destined to be outdated if there isn't a fresh supplyof real-world requirements and lessons learned being fed back as inputsfor the next go 'round. Innovation is a key step in the WorkforcePerformance Cycle. This step requires business to employ Stephen Covey'sfamous habit of “sharpening the saw”, or “take time to be fast”.

There is an untold wealth of information buried within the heads ofbusiness users responsible for implementing the steps outlined in theirpre-performance training and performance support tools. No other groupwithin an organization can more accurately assess the effectiveness ofcurrent methods, or project needed changes that will have dramaticfuture impact. This step of reflecting on the current and past state ofaffairs, uncovering new approaches by identifying what is working andwhat is not, and adapting accordingly for the future is the last phaseof the learning/performance cycle.

Innovation to fuel future training and performance support comesdirectly from the community most closely tied to task performance.Effective businesses need to develop and cultivate a mechanism forcommunication and collaboration among the experts in these communitiesto more efficiently benefit from their collective wisdom. In today'sbusiness, that which is evident to your business is evident to nearlyall your competitors as well. The competitive advantage lies inuncovering that which is unexpected or not immediately apparent,adapting your business processes to exploit the discovery, and quickly,but effectively, educating your workforce on the new policies andprocedures, all before the competition catches on or the market changesagain.

This innovation process is the critical final step in continuouseducation of the most effective and up-to-date policies, procedures, andinformation. Without formalized innovation, companies not only riskbeing a step behind the ever advancing competition, but compound theproblem by continuing to train their personnel with outdated strategiesand information. One way to formalize this vital step in the WorkforcePerformance Cycle is to construct Virtual Learning Communities, wheremany ‘experts’ can share experiences, submit ideas for improvements,play out “what-if” scenarios, and contribute on complex problems thatmay be insurmountable without significant collaboration with others.Such Learning Communities could nurture up-to-date discussion of what isactually happening within a business, eliminating the traditionalinformation-passing lag that plagues many business as new data travelsthrough corporate hierarchies. This increased nimbleness would helporganizations quickly address new competitive trends and outdatedstrategies, identify potential solutions, and implement improvedprocesses in the form of updated training and performance supportreference materials.

Currently, a typical BusSim engagement takes between one and two yearsto complete and requires a variety of both functional and technicalskills. FIG. 3 depicts the timeline and relative resource requirementsfor each phase of development for a typical application development inaccordance with a preferred embodiment. The chart clearly depicts therelationship between the large number of technical resources requiredfor both the build and test phases of development. This is because thetraditional development process used to build BusSim solutions reflectsmore of a “one off” philosophy, where development is done from scratchin a monolithic fashion, with little or no reuse from one application tothe next. This lack of reuse makes this approach prohibitivelyexpensive, as well as lengthy, for future BusSim projects.

The solution to this problem is to put tools in the hands ofinstructional designers that allows them to create their BusSim designsand implement them without the need for programmers to write code. Andto put application architectures that integrate with the tools in thehands of developers, providing them with the ability to quickly deliversolutions for a number of different platforms. The reuse, then, comes inusing the tools and architectures from one engagement to another. Bothfunctional and technical resources carry with them the knowledge of howto use the technology, which also has an associated benefit ofestablishing a best-practice development methodology for BusSimengagements.

The tools and architectures described herein are part of anext-generation Business Simulation delivery framework that will reducethe total effort necessary to build solutions in accordance with apreferred embodiment. FIG. 4 depicts the potential savings in bothfunctional and technical tasks in accordance with a preferredembodiment.

Development Cycle Activities

Design Phase

In the Design Phase, instructional designers become oriented to thecontent area and begin to conceptualize an instructional approach. Theyfamiliarize themselves with the subject matter through reading materialsand interviews with Subject Matter Experts (SMEs). They also identifylearning objectives from key client contacts. Conceptual designs forstudent interactions and interface layouts also begin to emerge. Afterthe conceptual designs have taken shape, Low-Fi user testing (a.k.a.Conference Room Piloting) is performed. Students interact with interfacemock-ups while facilitators observe and record any issues. Finally,detailed designs are created that incorporate findings. These detaileddesigns are handed off to the development team for implementation.

The design phase has traditionally been fraught with several problems.Unlike a traditional business system, BusSim solutions are not rooted intangible business processes, so requirements are difficult to identifyin a concrete way. This leaves instructional designers with a ‘blue sky’design problem. With few business-driven constraints on the solution,shallow expertise in the content area, and limited technical skills,instructional designers have little help in beginning a design.Typically, only experienced designers have been able to conjureinterface, analysis, and feedback designs that meet the learningobjectives yet remain technically feasible to implement. To compound theproblem, BusSim solutions are very open ended in nature. The designermust anticipate a huge combination of student behavior to designfeedback that is helpful and realistic.

Build Phase

During the build phase, the application development team uses thedetailed designs to code the application. Coding tasks include theinterfaces and widgets that the student interacts with. The interfacescan be made up of buttons, grids, check boxes, or any other screencontrols that allow the student to view and manipulate his deliverables.The developer must also code logic that analyzes the student's work andprovides feedback interactions. These interactions may take the form oftext and/or multimedia feedback from simulated team members,conversations with simulated team members, or direct manipulations ofthe student's work by simulated team members. In parallel with thesecoding efforts, graphics, videos, and audio are being created for use inthe application. Managing the development of these assets have their owncomplications.

Risks in the build phase include misinterpretation of the designs. Ifthe developer does not accurately understand the designer's intentions,the application will not function as desired.

Also, coding these applications requires very skilled developers becausethe logic that analyzes the student's work and composes feedback is verycomplex.

Test Phase

The Test Phase, as the name implies, is for testing the application.Testing is performed to verify the application in three ways: first thatthe application functions properly (functional testing), second that thestudents understand the interface and can navigate effectively(usability testing), and third that the learning objectives are met(cognition testing). Functional testing of the application can becarried out by the development team or by a dedicated test team. If theapplication fails to function properly, it is debugged, fixed,recompiled and retested until its operation is satisfactory. Usabilityand cognition testing can only be carried out by test students who areunfamiliar with the application. If usability is unsatisfactory, partsof the interface and or feedback logic may need to be redesigned,recoded, and retested. If the learning objectives are not met, largeparts of the application may need to be removed and completelyredeveloped from a different perspective.

The test phase is typically where most of the difficulties in the BusSimdevelopment cycle are encountered. The process of discovering and fixingfunctional, usability, and cognition problems is a difficult process andnot an exact science.

For functional testing, testers operate the application, either byfollowing a test script or by acting spontaneously and documenting theiractions as they go. When a problem or unexpected result is encountered,it too is documented. The application developer responsible for thatpart of the application then receives the documentation and attempts toduplicate the problem by repeating the tester's actions. When theproblem is duplicated, the developer investigates further to find thecause and implement a fix. The developer once again repeats the tester'sactions to verify that the fix solved the problem. Finally, all othertest scripts must be rerun to verify that the fix did not haveunintended consequences elsewhere in the application.

This process has inherent difficulty. If the tester is inaccurate inrecording his actions, or the developer is inaccurate in repeating them,then the problem may never be duplicated and the defect never found.Furthermore, the problem may be dependent on some condition in thetester's environment that is not readily observable or is not evenrelated to the application. This process has proven to be tedious,time-consuming, and of limited reliability.

For usability testing, test students operate the application as it willbe operated in production. Ideally, their progress is only impeded bytheir lack of mastery of the content. As they gain proficiency, they areable to complete the activities and move on. As is often the case,however, they are impeded by unclear instructions, a non-intuitiveinterface, or other usability shortcomings. When these difficulties areencountered, the facilitators document the problems and student commentson what is needed to improve usability.

There are two major risks associated with usability testing. First,student action recording is not as rigorous because actual students areperforming the testing, so functional problems that don't appear untilnow are difficult to reproduce. Second, resolutions to usabilityproblems often involve significant modification to the application codeand interface which requires repeating of portions of design, build, andtest.

For cognition testing, students are surveyed and/or tested to determinetheir level of understanding of the material. If results indicate thatthe learning objectives are not being adequately met, the design isreevaluated. Changes proposed to improve the cognition may includemassive redesign and rebuilding.

Execution Phase

The Execution Phase refers to the steady state operation of thecompleted application in its production environment. For some clients,this involves phone support for students. Clients may also want theability to track students” progress and control their progressionthrough the course. Lastly, clients may want the ability to track issuesso they may be considered for inclusion in course maintenance releases.

One of the key values of on-line courses is that they can be taken at atime, location, and pace that is convenient for the individual student.However, because students are not centrally located, support is notalways readily available. For this reason it is often desirable to havephone support for students.

Clients may also desire to track students” progress, or control theiradvancement through the course. Under this strategy, after a studentcompletes a section of the course, he will transfer his progress data toa processing center either electronically or by physically mailing adisk. There it can be analyzed to verify that he completed all requiredwork satisfactorily. One difficulty commonly associated with studenttracking is isolating the student data for analysis. It can be unwieldyto transmit all the course data, so it is often imperative to isolatethe minimum data required to perform the necessary analysis of thestudent's progress.

A Delivery Framework for Business Simulation

As discussed earlier, the traditional development process used to buildBusSim solutions reflects more of a “one off” philosophy, wheredevelopment is done from scratch in a monolithic fashion, with little orno reuse from one application to the next. A better approach would be tofocus on reducing the total effort required for development throughreuse, which, in turn would decrease cost and development time.

The first step in considering reuse as an option is the identificationof common aspects of the different BusSim applications that can begeneralized to be useful in future applications. In examination of theelements that make up these applications, three common aspects emerge asintegral parts of each:

Interface

Analysis

Interpretation

Interface

Every BusSim application must have a mechanism for interaction with thestudent. The degree of complexity of each interface may vary, from thehigh interactivity of a high-fidelity real-time simulation task, to theless complex information delivery requirements of a business casebackground information task. Regardless of how sophisticated the UserInterface (UI), it is a vital piece of making the underlying simulationand feedback logic useful to the end user.

Analysis

Every BusSim application does analysis on the data that defines thecurrent state of the simulation many times throughout the execution ofthe application. This analysis is done either to determine what ishappening in the simulation, or to perform additional calculations onthe data which are then fed back into the simulation. For example, theanalysis may be the recognition of any actions the student has taken onartifacts within the simulated environment (notebooks, number values,interviews conducted, etc.), or it may be the calculation of an ROIbased on numbers the student has supplied.

Interpretation

Substantive, useful feedback is a critical piece of any BusSimapplication. It is the main mechanism to communicate if actions taken bythe student are helping or hurting them meet their performanceobjectives. The interpretation piece of the set of proposed commonaltiestakes the results of any analysis performed and makes sense of it. Ittakes the non-biased view of the world that the Analysis portiondelivers (i.e., “Demand is up 3%”) and places some evaluative contextaround it (i.e., “Demand is below the expected 7%; you're in trouble!”,or “Demand has exceeded projections of 1.5%; Great job!”). FIG. 5illustrates commonalties in accordance with a preferred embodiment.

Common Features of Business Simulation Applications

There are several approaches to capturing commonalties for reuse. Two ofthe more common approaches are framework-based and component-based. Tohelp illustrate the differences between the two approaches, we will drawan analogy between building an application and building a house. One canconstruct a house from scratch, using the raw materials, 2×4s, nails,paint, concrete, etc. One can also construct an application fromscratch, using the raw materials of new designs and new code. The effortinvolved in both undertakings can be reduced through framework-basedand/or component-based reuse.

Framework-based Reuse

Within the paradigm of framework-based reuse, a generic framework orarchitecture is constructed that contains commonalties. In the houseanalogy, one could purchase a prefabricated house framework consistingof floors, outside walls, bearing walls and a roof The house can becustomized by adding partition walls, wall-paper, woodwork, carpetingetc. Similarly, prefabricated application frameworks are available thatcontain baseline application structure and functionality. Individualapplications are completed by adding specific functionality andcustomizing the look-and-feel. An example of a commonly used applicationframework is Microsoft Foundation Classes. It is a framework fordeveloping Windows applications using C++. MFC supplies the basefunctionality of a windowing application and the developer completes theapplication by adding functionality within the framework.Framework-based reuse is best suited for capturing template-likefeatures, for example user interface management, procedural objectbehaviors, and any other features that may require specialization.

Some benefits of using a framework include:

Extensive functionality can be incorporated into a framework. In thehouse analogy, if I know I am going to build a whole neighborhood ofthree bedroom ranches, I can build the plumbing, wiring, and partitionwalls right into the framework, reducing the incremental effort requiredfor each house. If I know I am going to build a large number of verysimilar applications, they will have more commonalties that can beincluded in the framework rather than built individually.

Applications can override the framework-supplied functionality whereverappropriate. If a house framework came with pre-painted walls, thebuilder could just paint over them with preferred colors. Similarly, theobject oriented principle of inheritance allows an application developerto override the behavior of the framework.

Component-based Reuse

In the paradigm of component-based reuse, key functionality isencapsulated in a component. The component can then be reused inmultiple applications. In the house analogy, components correspond toappliances such as dishwashers, refrigerators, microwaves, etc.Similarly, many application components with pre-packaged functionalityare available from a variety of vendors. An example of a popularcomponent is a Data Grid. It is a component that can be integrated intoan application to deliver the capability of viewing columnar data in aspreadsheet-like grid. Component-based reuse is best suited forcapturing black-box-like features, for example text processing, datamanipulation, or any other features that do not require specialization.

Some benefits of using components include:

Several applications on the same computer can share a single component.This is not such a good fit with the analogy, but imagine if all thehouses in a neighborhood could share the same dishwasher simultaneously.Each home would have to supply its own dishes, detergent, and water, butthey could all wash dishes in parallel. In the application componentworld, this type of sharing is easily accomplished and results inreduced disk and memory requirements.

Components tend to be less platform and tool dependent. A microwave canbe used in virtually any house, whether it's framework is steel or wood,and regardless of whether it was customized for building mansions orshacks. You can put a high-end microwave in a low-end house andvice-versa. You can even have multiple different microwaves in yourhouse. Component technologies such as CORBA, COM, and Java Beans makethis kind of flexibility commonplace in application development.

The Solution: A Combined Approach

Often, the best answer to achieving reuse is through a combination offramework-based and component-based techniques. A framework-basedapproach for building BusSim applications is appropriate for developingthe user interface, handling user and system events, starting andstopping the application, and other application-specific and deliveryplatform-specific functions. A component-based approach is appropriatefor black-box functionality. That is, functionality that can be usedas-is with no specialization required.

In creating architectures to support BusSim application development, itis imperative that any assets remain as flexible and extensible aspossible or reusability may be diminished. Therefore, we chose toimplement the unique aspects of BusSim applications using a componentapproach rather than a framework approach. This decision is furthersupported by the following observations.

An application can only be based on one framework. Using the houseanalogy, if you like the first floor of one framework and the secondfloor of another, it is difficult or impossible to integrate thefeatures of the two. Or, it is so costly as to erase the benefit ofusing a framework in the first place. Likewise with applicationframeworks. You can only use one framework when building an applicationYou can't mix and match features from multiple frameworks, so anyframework that we developed would have to compete against existing andfuture frameworks. With components, however, you can mix and match frommultiple vendors.

Components are less platform and development tool dependent, leavingmore options open for development teams. An appliance like a dishwasheris not restricted for use in a particular type of house. Similarly,component technologies exist that are independent of platform anddevelopment tool. For example ActiveX can be used in almost everydevelopment environment for Windows and Java Beans components can beused on a wide variety of platforms.

Frameworks become obsolete more quickly. Rapid emergence and evolutionof technology has introduced a wealth of new feature requirements intoapplication development. Frameworks that do not include the most currentfeatures become obsolete quickly. Components typically address a morefocused feature set and are not as impacted by technology advancesoutside their core functionality areas.

Based on these observations, we believe a combined framework/componentapproach is optimal for achieving reuse. FIG. 6 illustrates adevelopment architecture approach in accordance with a preferredembodiment.

Delivery Framework for Business Simulation

This diagram illustrates an ideal solution where components are combinedwith an Application Framework and an Application Architecture to achievemaximum reuse and minimum custom development effort. The ApplicationArchitecture is added to provide communication support between theapplication interface and the components, and between the components.This solution has the following features:

The components (identified by the icons) encapsulate key BusSimfunctionality.

The Application Architecture provides the glue that allowsapplication-to-component and component-to-component communication.

The Application Framework provides structure and base functionality thatcan be customized for different interaction styles.

Only the application interface must be custom developed.

The next section discusses each of these components in further detail.

The Business Simulation Toolset

We have clearly defined why a combined component/framework approach isthe best solution for delivering high-quality BusSim solutions at alower cost. Given that there are a number of third party frameworksalready on the market that provide delivery capability for a widevariety of platforms, the TEL project is focused on defining anddeveloping a set of components that provide unique services for thedevelopment and delivery of BusSim solutions. These components alongwith a set of design and test workbenches are the tools used byinstructional designers to support activities in the four phases ofBusSim development. We call this suite of tools the Business SimulationToolset. Following is a description of each of the components andworkbenches of the toolset.

Components

A Component can be thought of as a black box that encapsulates thebehavior and data necessary to support a related set of services. Itexposes these services to the outside world through publishedinterfaces. The published interface of a component allows you tounderstand what it does through the services it offers, but not how itdoes it. The complexity of its implementation is hidden from the user.The following are the key components of the BusSim Toolset.

Domain Component—provides services for modeling the state of asimulation

Profiling Component—provides services for rule-based evaluating thestate of a simulation

Transformation Component—provides services for manipulating the state ofa simulation

Remediation Component—provides services for the rule-based delivering offeedback to the student.

Domain Component

The Domain Model component is the central component of the suite thatfacilitates communication of context data across the application and theother components. It is a modeling tool that can use industry-standarddatabase such as Informix, Oracle, or Sybase to store its data. A domainmodel is a representation of the objects in a simulation. The objectsare such pseudo tangible things as a lever the student can pull, a formor notepad the student fills out, a character the student interacts within a simulated meeting, etc. They can also be abstract objects such asthe ROI for a particular investment, the number of times the studentasked a particular question, etc. These objects are called entities.Some example entities include:

Vehicles, operators and incidents in an insurance domain

Journal entries, cash flow statements and balance sheets in a financialaccounting domain

Consumers and purchases in a marketing domain

An entity can also contain other entities. For example, a personal bankaccount entity might contain an entity that represents a savings accountEvery entity has a set of properties where each property in some waydescribes the entity. The set of properties owned by an entity, inessence, define the entity. Some example properties include:

An incident entity on an insurance application owns properties such as“Occurrence Date”, “Incident Type Code”, etc.

A journal entry owns properties such as “Credit Account”, “DebitAccount”, and “Amount”

A revolving credit account entity on a mortgage application ownsproperties such as “Outstanding Balance”, “Available Limit”, etc. FIG. 7illustrates a small segment of a domain model for claims handlers in theauto insurance industry in accordance with a preferred embodiment.

Example Domain Model

The domain model is created by the instructional designer in a visualediting design tool called the Knowledge Workbench. The designer createsthe objects of the domain model using generic entities and properties;that is, not having specific values associated with the entities andproperties.

At runtime, an application's domain model is instantiated so that everyentity and property is given a particular value that makes it unique.The result of a domain model instantiation is called the domain. Thevalues of a domain's entities and properties can change throughout thecourse of the simulation based on student actions and updates from othercomponents. FIG. 8 illustrates an instantiated domain model inaccordance with a preferred embodiment.

Example Domain

Creating a domain model in data rather than in code facilitates reuse ofthe components in multiple applications in multiple domains without codechanges. For example, a typical application in the Financial Servicesdomain would have to define classes in code such as ‘Customer’,‘Account’, etc. An Insurance Domain application might have classes suchas ‘Customer’, ‘Incident’, ‘Prior Policy’, etc. To be able to performanalysis on any of these classes, the analysis logic must be coded torecognize the classes. This requires all logic to be custom-coded forevery application; an effort-intensive undertaking that demands a highdegree of technical skill.

By modeling the domain in data using generic objects, we can buildstandard generic analysis capability that can be applied to the domain.This allows implementation of analysis logic with much less effort bypeople with a low degree of technical skill. Functional experts cancreate the objects of the domain and apply various types of analysisfrom a pallet. All of this is accomplished in a visual developmentenvironment that supports the designer with visual feedback and onlyallows valid designs to be created.

Profiling Component

In the simplest terms, the purpose of the Profiling Component is toanalyze the current state of a domain and identify specific things thatare true about that domain. This information is then passed to theRemediation Component which provides feedback to the student. TheProfiling Component analyzes the domain by asking questions about thedomain's state, akin to an investigator asking questions about a case.The questions that the Profiler asks are called profiles. For example,suppose there is a task about building a campfire and the student hasjust thrown a match on a pile of wood, but the fire didn't start. Inorder to give useful feedback to the student, a tutor would need to knowthings like: was the match lit?, was the wood wet?, was there kindlingin the pile?, etc. These questions would be among the profiles that theProfiling Component would use to analyze the domain. The results of theanalysis would then be passed off to the Remediation Component whichwould use this information to provide specific feedback to the student.

Specifically, a profile is a set of criteria that is matched against thedomain. The purpose of a profile is to check whether the criteriadefined by the profile is met in the domain. Using a visual editingtool, instructional designers create profiles to identify those thingsthat are important to know about the domain for a given task. Duringexecution of a BusSim application at the point that feedback isrequested either by the student or pro-actively by the application, theset of profiles associated with the current task are evaluated todetermine which ones are true. Example profiles include:

Good productions strategy but wrong Break-Even Formula

Good driving record and low claims history

Correct Cash Flow Analysis but poor Return on Investment (ROI)

A profile is composed of two types of structures: characteristics andcollective characteristics. A characteristic is a conditional (the ifhalf of a rule) that identifies a subset of the domain that is importantfor determining what feedback to deliver to the student. Examplecharacteristics include:

Wrong debit account in transaction 1

Perfect cost classification

At Least 1 DUI in the last 3 years

More than $4000 in claims in the last 2 years

More than two at-fault accidents in 5 years

A characteristic's conditional uses one or more atomics as the operandsto identify the subset of the domain that defines the characteristic. Anatomic only makes reference to a single property of a single entity inthe domain; thus the term atomic. Example atomics include:

The number of DUI's>=1

ROI>10%

Income between $75,000 and $110,000

A collective characteristic is a conditional that uses multiplecharacteristics and/or other collective characteristics as its operands.Collective characteristics allow instructional designers to build richerexpressions (i.e., ask more complex questions). Example collectivecharacteristics include:

Bad Household driving record

Good Credit Rating

Marginal Credit Rating

Problems with Cash for Expense transactions

Problems with Sources and uses of cash

Once created, designers are able to reuse these elements within multipleexpressions, which significantly eases the burden of creating additionalprofiles. When building a profile from its elements, atomics can be usedby multiple characteristics, characteristics can be used by multiplecollective characteristics and profiles, and collective characteristicscan be used by multiple collective characteristics and profiles. FIG. 9illustrates an insurance underwriting profile in accordance with apreferred embodiment.

Example Profile for Insurance Underwriting

Transformation Component

Whereas the Profiling Component asks questions about the domain, theTransformation Component performs calculations on the domain and feedsthe results back into the domain for further analysis by the ProfilingComponent. This facilitates the modeling of complex business systemsthat would otherwise be very difficult to implement as part of theapplication. Within the Analysis phase of theInterface/Analysis/Interpretation execution flow, the TransformationComponent actually acts on the domain before the Profiling Componentdoes its analysis. The Transformation Component acts as a shell thatwraps one or more data modeling components for the purpose ofintegrating these components into a BusSim application. TheTransformation Component facilitates the transfer of specific data fromthe domain to the data modeling component (inputs) for calculations tobe performed on the data, as well as the transfer of the results of thecalculations from the data modeling component back to the domain(outputs). FIG. 10 illustrates a transformation component in accordancewith a preferred embodiment.

The data modeling components could be third party modeling environmentssuch as spreadsheet-based modeling (e.g., Excel, Formula 1) or discretetime-based simulation modeling (e.g., PowerSim, VenSim). The componentscould also be custom built in C++, VB, Access, or any tool that is ODBCcompliant to provide unique modeling environments. Using theTransformation Component to wrap a third party spreadsheet componentprovides an easy way of integrating into an applicationspreadsheet-based data analysis, created by such tools as Excel. TheTransformation Component provides a shell for the spreadsheet so that itcan look into the domain, pull out values needed as inputs, performs itscalculations, and post outputs back to the domain.

For example, if the financial statements of a company are stored in thedomain, the domain would hold the baseline data like how much cash thecompany has, what its assets and liabilities are, etc. TheTransformation Component would be able to look at the data and calculateadditional values like cash flow ratios, ROI or NPV of investments, orany other calculations to quantitatively analyze the financial health ofthe company. Depending on their complexity, these calculations could beperformed by pre-existing spreadsheets that a client has already spentconsiderable time developing.

Remediation Component

The Remediation Component is an expert system that facilitatesintegration of intelligent feedback into BusSim applications. It has thefollowing features:

Ability to compose high quality text feedback.

Ability to compose multimedia feedback that includes video and/or audio.

Ability to include reference material in feedback such as Authorwarepages or Web Pages.

Ability to actively manipulate the users deliverables to highlight oreven fix users' errors.

A proven remediation theory embedded in its feedback compositionalgorithm.

Allows integration of digital assets into the Remediation of a trainingor IPS application.

The Remediation model consists of three primary objects:

Concepts

Coach Topics

Coach Items

Concepts are objects that represent real-world concepts that the userwill be faced with in the interface. Concepts can be broken intosub-concepts, creating a hierarchical tree of concepts.

This tree can be arbitrarily deep and wide to support rich conceptmodeling. Concepts can also own an arbitrary number of Coach Topics.

Coach Topics are objects that represent a discussion topic that may beappropriate for a concept. Coach Topics can own an arbitrary number ofCoach Items.

Coach Items are items of feedback that may include text, audio, video,URL's, or updates to the Domain Model. Coach Items are owned by CoachTopics and are assembled by the Remediation Component algorithm. Detailsof the Remediation Component algorithm for feedback is discussed in TheIntelligent

Coaching Agent Tool whitepaper and can be found on the KnowledgeExchange at the Technology Enabled Learning ETA Home Page.

Workbenches

The BusSim Toolset also includes a set of workbenches that are used byinstructional designers to design and build BusSim applications. Aworkbench is a tool that facilitates visual editing or testing of thedata that the BusSim Components use for determining an application'srun-time behavior. The BusSim Toolset includes the followingworkbenches:

Knowledge Workbench

The Knowledge Workbench is a tool for the creation of domain, analysisand feedback data that is used by the BusSim Components. It has thefollowing features:

Allows the designer to ‘paint’ knowledge in a drag-and-drop interface.

Knowledge is represented visually for easy communication amongdesigners.

The interface is intelligent, allowing designers to only paint validinteractions.

Designer's Task creations are stored in a central repository.

The workbench supports check-in/check-out for exclusive editing of atask.

Supports LAN-based or untethered editing.

Automatically generates documentation of the designs.

Generates the data files that drive the behavior of the components.

Simulated Student Test Workbench

The Simulated Student Test Workbench is a tool for the creation of datathat simulates student's actions for testing BusSim Component behaviors.It has the following features:

The Test Bench generates a simulated application interface based on theDomain Model.

The designer manipulates the objects in the Domain Model to simulatestudent activity.

The designer can invoke the components to experience the interactionsthe student will experience in production.

The designer can fully test the interaction behavior prior todevelopment of the application interface.

Regression Test Workbench

The Regression Test Workbench is a tool for replaying and testing ofstudent sessions to aid debugging. It has the following features:

Each student submission can be individually replayed through thecomponents.

An arbitrary number of student submissions from the same session can bereplayed in succession.

Entire student sessions can be replayed in batch instantly.

The interaction results of the student are juxtaposed with the resultsof the regression test for comparison.

Development Cycle Activities

Design Phase

The design phase of a BusSim application is streamlined by the use ofthe Knowledge Workbench. The Knowledge Workbench is a visual editor forconfiguring the objects of the component engines to control theirruntime behavior. The components are based on proven algorithms thatcapture and implement best practices and provide a conceptual frameworkand methodology for instructional design.

In conceptual design, the workbench allows the designer to paint a modelof the hierarchy of Concepts that the student will need to master in theactivity. This helps the designer organize the content in a logical way.The visual representation of the Concepts helps to communicate ideas toother designers for review. The consistent look and feel of theworkbench also contributes to a streamlined Quality Assurance process.In addition, standard documentation can be automatically generated forthe entire design.

As the design phase progresses, the designer adds more detail to thedesign of the Concept hierarchy by painting in Coach Topics that thestudent may need feedback on. The designer can associate multiplefeedback topics with each Concept. The designer also characterizes eachtopic as being Praise, Polish, Focus, Redirect or one of several othertypes of feedback that are consistent with a proven remediationmethodology. The designer can then fill each topic with text, video warstories, Web page links, Authorware links, or any other media objectthat can be delivered to the student as part of the feedback topic.

As the designer's thoughts for the interface become clearer, she canbegin to model the domain objects in the Knowledge Workbench. Thestudent's world is constructed using objects in the Domain Model.

The designer again uses the Knowledge Workbench to configure objects inthe Transformation Component. The Transformation Component is used toperform calculations or other analysis of the student's domain. Lastly,the designer uses the workbench to configure objects in the ProfilingComponent. The Profiling Component examines the student's domain,looking for conditions that indicate what feedback topics areappropriate for delivery.

More importantly, the Student Simulator Test Workbench allows thedesigner to exercise the designs. It allows the designer to manipulatethe domain as if she were a student. The designer can interact with thesimulated interface and invoke the component engines to see the feedbackthat the student would receive. This capability can also be utilized ina usability test such as a Conference Room Pilot. As the test studentinteracts with screen mock-ups, a facilitator can mimic his actions inthe interface simulator and tell the student what the actual feedbackwill be. This results in much more rigorous testing prior to applicationconstruction. A big payoff is realized downstream in the form of reducedredesign after usability and cognition testing.

Throughout all these steps in the initial design, the workbench supportsthe design process by allowing the designer great flexibility within theframework of a proven methodology. This allows experienced users todesign rich, realistic interactions, and inexperienced users to becomecompetent in a shorter time by learning from the best practices embeddedin the workbench. This greatly diminishes the ‘blue sky’ design problem.Also, since the designs can be tested prior to application construction,there is reduced rework after testing. Lastly, the visual knowledgerepresentation enhances communication within the design team and greatlystreamlines the QA process.

Build Phase

It is very clear how the tools support the Build Phase. The designs thatthe designer painted in the Knowledge Workbench drive the components atruntime. The application developer no longer has to write code thatanalyzes the student's work and provides feedback. The developer onlyhas to build the interface and logic to report any student actions tothe domain model. The components do the rest. What used to be the mostdifficult part of the build phase has been eliminated!

There is no chance for a developer to misinterpret the feedback designsbecause she never has to interpret them. In fact, the developer doesn'teven have to know anything about the feedback behavior as long as she isfamiliar with the domain model. This also means the skill level requiredto develop the application can be greatly reduced. It's not hard toteach someone how to paint a screen in Visual Basic or Delphi and callAPI functions to notify the Domain Model of student actions.

In addition to the economies gained by the components, it is possible touse templates to further streamline design and development of commonlyused interactions. We have created templates for several commoninteractions. For example, Journalizing of Transactions is aninteraction that has appeared in several applications. We have builtapplication and Knowledge Workbench templates for Journalization. Allone must do to create a new Journalize task is to add graphics for newTransactions and fill in new data into placeholders in the KnowledgeWorkbench.

Test Phase

The toolset greatly reduces effort during functionality testing. The keydriver of the effort reduction is that the components can automaticallytrack the actions of the tester without the need to add code support inthe application. Whenever the tester takes an action in the interface,it is reported to the domain model. From there it can be tracked in adatabase. Testers no longer need to write down their actions for use indebugging; they are automatically written to disk. There is also afeature for attaching comments to a tester's actions. When unexpectedbehavior is encountered, the tester can hit a control key sequence thatpops up a dialog to record a description of the errant behavior.

Of far greater impact is the ability to replay the tester's actionsautomatically through the Regression Test Workbench. The designer doesnot need to spend hours trying to duplicate the error. She simply loadsthe tester's session into the Regression Test Workbench and replays it.In seconds the error is replicated and can be located and fixed using avariety of debugging utilities. After changes have been made, one morepass through the Regression Test Workbench verifies the fix.

The major difficulties of usability and cognition testing are alsoaddressed by the toolset. First, since student tracking is no longer amanual activity, the precision of functional testing can also be appliedto usability and cognition testing. Second, because of the increasedrigor in the Conference Room Pilot, the risk of significant rework isgreatly reduced.

Execution Phase

During the Execution Phase, the components are deployed to the student'splatform. They provide simulated team member and feedback functionalitywith sub-second response time and error-free operation. If the clientdesires it, student tracking mechanisms can be deployed at runtime forevaluation and administration of students. This also enables theisolation of any defects that may have made it to production.

Scenarios for Using the Business Simulation Toolset

A good way to gain a better appreciation for how the BusSim Toolset canvastly improve the BusSim development effort is to walk throughscenarios of how the tools would be used throughout the developmentlifecycle of a particular task in a BusSim application. For thispurpose, we'll assume that the goal of the student in a specific task isto journalize invoice transactions, and that this task is within thebroader context of learning the fundamentals of financial accounting. Acursory description of the task from the student's perspective will helpset the context for the scenarios. Following the description are fivescenarios which describe various activities in the development of thistask. The figure below shows a screen shot of the task interface.

FIG. 11 illustrates the use of a toolbar to navigate and accessapplication level features in accordance with a preferred embodiment. Astudent uses a toolbar to navigate and also to access some of theapplication-level features of the application. The toolbar is theinverted L-shaped object across the top and left of the interface. Thetop section of the toolbar allows the user to navigate to tasks withinthe current activity. The left section of the toolbar allows the studentto access other features of the application, including feedback. Thestudent can have his deliverables analyzed and receive feedback byclicking on the Team button. In this task, the student must journalizetwenty-two invoices and other source documents to record the flow ofbudget dollars between internal accounts. (Note: “Journalizing”, or“Journalization”, is the process of recording journal entries in ageneral ledger from invoices or other source documents during anaccounting period. The process entails creating debit and balancingcredit entries for each document. At the completion of this process, thegeneral ledger records are used to create a trial balance and subsequentfinancial reports.)

In accordance with a preferred embodiment, an Intelligent Coaching AgentTool (ICAT) was developed to standardize and simplify the creation anddelivery of feedback in a highly complex and open-ended environment.Feedback from a coach or tutor is instrumental in guiding the learnerthrough an application. Moreover, by diagnosing trouble areas andrecommending specific actions based on predicted student understandingof the domain student comprehension of key concepts is increased. Bywriting rules and feedback that correspond to a proven feedbackstrategy, consistent feedback is delivered throughout the application,regardless of the interaction type or of the specific designer/developercreating the feedback. The ICAT is packaged with a user-friendlyworkbench, so that it may be reused to increase productivity on projectsrequiring a similar rule-based data engine and repository.

Definition of ICAT In Accordance With a Preferred Embodiment

The Intelligent Coaching Agent Tool (ICAT) is a suite of tools—adatabase and a Dynamic Link Library (DLL) run-time engine—used bydesigners to create and execute just-in-time feedback of Goal Basedtraining. Designers write feedback and rules in the development tools.Once the feedback is set, the run-time engine monitors user actions,fires rules and composes feedback which describes the businessdeliverable.

I. The ICAT Remediation Model

The remediation model used within ICAT dynamically composes the mostappropriate feedback to deliver to a student based on student's previousresponses. The ICAT model is based on a theory of feedback which hasbeen proven effective by pilot results and informal interviews. Themodel is embodied in the object model and algorithms of the ICAT.Because the model is built into the tools, all feedback created with thetool will conform to the model.

II. The Role of ICAT in Student Training

The ICAT plays two roles in student training. First, the ICAT is ateaching system, helping students to fully comprehend and applyinformation. Second, ICAT is a gatekeeper, ensuring that each studenthas mastered the material before moving on to additional information.

III. The Functional Definition of the ICAT

The ICAT is a self contained module, separate from the application.Separating the ICAT from the application allows other projects to usethe ICAT and allows designers to test feedback before the application iscomplete. The ICAT Module is built on six processes which allow astudent to interact effectively with the interface to compose anddeliver the appropriate feedback for a student's mistakes.

IV. The ICAT Development Methodology for Creating Feedback

The ICAT development methodology is a seven step methodology forcreating feedback. The methodology contains specific steps, generalguidelines and lessons learned from the field. Using the methodologyincreases the effectiveness of the feedback to meet the educationalrequirements of the course.

V. Components

The processes each contain a knowledge model and some containalgorithms. Each process has specific knowledge architected into itsdesign to enhance remediation and teaching.

VI. Testing Utilities, Reports and Methodology

There is a suite of testing tools for the ICAT. These tools allowdesigners and developers test all of their feedback and rules. Inaddition, the utilities let designers capture real time activities ofstudents as they go through the course.

Expert Remediation Model Within the Tools

The tools and run-time engine in accordance with a preferred embodimentinclude expert knowledge of remediation. These objects include logicthat analyzes a student's work to identify problem areas and deliverfocused feedback. The designers need only instantiate the objects to putthe tools to work. Embodying expert knowledge in the tools and engineensures that each section of a course has the same effective feedbackstructure in place.

Any project which is creating a Goal-Based Scenario (GBS) businesssimulation or an Integrated Performance Support (IPS) system to helpusers understand and create business deliverables can profit fromtechnology in accordance with-a preferred embodiment. A GBS allowsstudents to learn in a comprehensive simulated environment. Studentswork in a simulated environment to accomplish real world tasks, and whenthey make mistakes, remediation is provided to help identify and correctthe mistakes. The hands-on experience of the simulated environment andthe timely remediation account for the high retention rate from subjectspresented utilizing a system in accordance with a preferred embodiment.A system in accordance with a preferred embodiment can be used inconjunction with an IPS to help users develop deliverables. If acustomer service representative (CSR) is completing a form whileconducting a phone conversation, the ICAT can be used to observe how thetask is completed to provide a live analysis of mistakes.

A file structure in accordance with a preferred embodiment provides astandard system environment for all applications in accordance with apreferred embodiment. A development directory holds a plurality ofsub-directories. The content in the documentation directory is part of aseparate installation from the architecture. This is due to the size ofthe documentation directory. It does not require any support files, thusit may be placed on a LAN or on individual computers.

When the architecture is installed in accordance with a preferredembodiment, the development directory has an _Arch, _Tools, _Utilities,Documentation, QED, and XDefault development directory. Each folder hasits own directory structure that is inter-linked with the otherdirectories. This structure must be maintained to assure consistency andcompatibility between projects to clarify project differences, andarchitecture updates.

The _Arch directory stores many of the most common parts of the systemarchitecture. These files generally do not change and can be reused inany area of the project. If there is common visual basic code forapplications that will continuously be used in other applications, thefiles will be housed in a folder in this directory.

The sub-directories in the _Arch directory are broken into certainobjects of the main project. Object in this case refers to parts of aproject that are commonly referred to within the project. For example,modules and classes are defined here, and the directory is analogous toa library of functions, APIs, etc . . . that do not change. For examplethe IcaObj directory stores code for the Intelligent Coaching Agent(ICA). The InBoxObj directory stores code for the InBox part of theproject and so on. The file structure uses some primary objectreferences as file directories. For example, the IcaObj directory is acomponent that contains primary objects for the ICA such as functionalforms, modules and classes.

The BrowserObj directory contains modules, classes and forms related tothe browser functionality in the architecture.

The HTMLGlossary directory contains code that is used for the HTMLreference and glossary component of the architecture.

The IcaObj directory contains ICA functional code to be used in anapplication. This code is instantiated and enhanced in accordance with apreferred embodiment.

The InBoxObj directory contains code pertaining to the inboxfunctionality used within the architecture. Specifically, there are twomajor components in this architecture directory. There is a new .ocxcontrol that was created to provide functionality for an inbox in theapplication. There is also code that provides support for a legacy inboxapplication. The PracticeObj directory contains code for the topicscomponent of the architecture. The topics component can be implementedwith the HTMLGlossary component as well.

The QmediaObj directory contains the components that are media related.An example is the QVIDctrl.cls. The QVIDctrl is the code that createsthe links between QVID files in an application and the system inaccordance with a preferred embodiment.

The SimObj directory contains the Simulation Engine, a component of theapplication that notifies the tutor of inputs and outputs using aspreadsheet to facilitate communication.

The StaticObj directory holds any component that the application willuse statically from the rest of the application. For example, the loginform is kept in this folder and is used as a static object in accordancewith a preferred embodiment.

The SysDynObj directory contains the code that allows the SystemsDynamics Engine (Powersim) to pass values to the Simulation Engine andreturn the values to the tutor.

The VBObj directory contains common Visual Basic objects used inapplications. For example the NowWhat, Visual Basic Reference forms, andspecific message box components are stored in this folder.

The _Tools directory contains two main directories. They represent thetwo most used tools in accordance with a preferred embodiment. The twodirectories provide the code for the tools themselves. The reason forproviding the code for these tools is to allow a developer to enhancecertain parts of the tools to extend their ability. This is importantfor the current project development and also for the growth of thetools.

The Icautils directory contains a data, database, default, graphics,icadoc, and testdata directory. The purpose of all of these directoriesis to provide a secondary working directory for a developer to keeptheir testing environment of enhanced Icautils applications separatefrom the project application. It is built as a testbed for the toolonly. No application specific work should be done here. The purpose ofeach of these directories will be explained in more depth in the projectdirectory section. The TestData folder is unique to the _Tools/ICAUtilsdirectory. It contains test data for the regression bench among otherscomponents in ICAUtils.

Utilities

The Utilities directory holds the available utilities that a BusinessSimulation project requires for optimal results. This is a repositoryfor code and executable utilities that developers and designers mayutilize and enhance in accordance with a preferred embodiment. Most ofthe utilities are small applications or tools that can be used in theproduction of simulations which comprise an executable and code to gowith it for any enhancements or changes to the utility. If new utilitiesare created on a project or existing utilities are enhanced, it isimportant to notify the managers or developers in charge of keepingtrack of the Business Simulation assets. Any enhancements, changes oradditions to the Business Simulation technology assets are important forfuture and existing projects.

Documentation

A Documentation directory is used to store pertinent documentation. Thedocumentation directory is structured as follows. Most of thedirectories are labeled after the specific information held within them.The following is a list of all the documentation directories and adescription of what is contain in each.

Ref Website—This directory contains The Business Simulation Referencewebsite, which is a general reference for many things. If the websitehas not been set up for users on a LAN or website, all you need to do isgo into the root directory of website and double click on index.htm.This is the main page for the site.

Components—This directory contains any documentation on classes andmodules that are used in the archtecture. For example there aredocuments here on the ICAMeeting class, the Inbox class etc.

Database—This directory contains any documents describing the databasesthat are included and used in the Architecture. For example the ICAObjoverview doc contains a description of the model and each element in thedatabase.

HTML Component—This directory contains relevant documentation about theHTML part of the architecture.

Process Models—This directory should contain the documents that describethe process of the application or related information.

ReferenceApp—This directory contains documents with descriptions andviews of the reference app. (QED) for explanation and documentation.Testing conditions are stored in the Testing directory.

Standards&Templates—This directory contains any type of architecturerelevant coding standard documents or templates that a developer isrequired to follow.

UserGuides—This directory has 6 sub-directories. Each one of thesesub-directories contains user guides for a given tool or component inaccordance with a preferred embodiment which include user guides for thearchitecture, the Tutor Suite, ICA Utilities, the simulation Engine andthe System Dynamics Engine. There is also a directory for otherdocumentation that contains user guides for any other tools or code likethird party controls etc.

WorkFlows—This directory contains the WF_Develop.doc which includes theworkflow documentation for an application.

Project Directory

The sample project directory, QED has the same structure that a realproject would be designed after. The QED directory has all customarchitecture code, databases, spreadsheets, and any other applicationspecific files stored in it. The QED project directory stores a Designand SrcVB directory. The Design directory contains all relevant filesfor a designer. The SrcVB directory is used for a developer.

The root directory of the Design and SrcVB directory contain a fewimportant files to note. Both have two .rtf files, a few log files andan .ini file. The .rtf files are the feedback that is output from thetutor, the logs are also output from the tutor and the .ini file is forICAUtils initialization. The design directory has three subdirectoriesthat contain a data directory, which stores .xls files, sim models, andany other important data like html and video. It also has a databasedirectory that holds any relevant databases for development andapplication use. The last directory is the icadoc directory whichincludes all tut files or .ica files, which are both created with thetutor.

The SrcVB directory stores all of the directories previously described.The reason for duplicating the data and database directories is toassure that a developer does not interfere with the designer's files.The developer tends to not do as much design work and can easily corruptfiles. This duplication of directories provides a safer environment forthe developer to test in. As was mentioned above, the developer tends tohave a lot more to do with the application build than the design sothere needs to be more content in the SrcVB directory. The SrcVBdirectory also contains an .exe and .vbp file which are created in adevelopers visual basic application.

The following are directories found in the SrcVB directory that are notfound in the Design directory followed by a short definition:

The _CustomArch directory contains any application specificarchitecture. Look in the QED folder for an example.

The _CustomDistribution directory contains any files that need to bedistributed with the application.

The Default directory contains any backup files that might need to becopied and reused later. Some files occasionally are corrupted and needto be replaced.

The Fonts directory contains application specific font libraries.

The Graphics directory contains any relevant graphics for theapplication.

The Help directory contains all files for a help reference layer in theapplication. This can be implemented in many ways but most commonly inan HTML form.

The Saved directory is for saved information that is produced by theapplication. This can be used for saving student information or savingtemporary information for the application steps.

The StudentData directory is for storing any relevant student data,lists of students, their personal information or any relevant studentdata that needs to be saved.

XDefault Development:

The XDefault Development environment is used to provide a shell for anynew project. A developer would rename this directory as an acronym ofthe project. QED is the default for the installation sample application.The XDefault development directory is a shell and serves as a buildingblock for a startup project. A good idea is to use the QED sampleapplication and build the XDefault Development project with the samplecode in QED.

Shared Development

The last directory to be mentioned is the shared development directorywhich is placed on a LAN or central network area and is shared by alldesigners and developers of a project to assure that files in theproject are up to date, managed properly and appropriately synchronized.There are many databases and files that will be shared in accordancewith a preferred embodiment. These files need to be shared and have alocation that someone can edit without having to worry about mergingfiles later. A source control program is used to restrict access to afile to one application at a time.

The ICAT Model of Remediation

The ICAT has a model of remediation architected into the system inaccordance with a preferred embodiment. Feedback should help studentscomplete tasks and learn the underlying concepts. To achieve this goal,the ICAT reviews student's work with the following objectives in mind.

Identify Student Misconceptions

Identifying that a student does not understand a topic and then clearlyexplaining it is the goal of human and computer tutors alike. Humantutors, however, have many more clues—including facial expressions andbody language—to help them identify student misconceptions. The computertutor is much more limited and can only view the outputs—such asdocuments and reports—the student produces. If a computer tutor islooking for a misunderstanding about debits and credits, the computeranalyzes all the mistakes a student made concerning debits and creditsand tries to identify what misunderstanding would account for thispattern of mistakes.

Identify What Students Should Fix

If the coach cannot diagnose a student's misconception, or cannot do itwith 100% accuracy, the coach must at least tell the student what he didwrong so that he can correct it. If at all possible, the coach shouldidentify groups or types of problems the student is making so that thestudent can generalize the solution and answer.

Prompt Students to Reflect on Mistakes

When identifying problems, the tutor needs to prompt the student toreflect on a problem and start to point the student towards the answer.The tutor should not tell the student the answer, but instead shouldattempt to provide an appropriate answer or give the student a questionto think about.

Reinforce Correct Concepts and Ideas

Once a student has gotten the correct answer, it is important toreinforce the learning. Students may feel uncertain about theirunderstanding even after he has gotten the answer correct. To reinforcethe student's understanding of the concept and provide a confidenceboost, the tutor should walk the student through the answer so that itis completely understood. These goals are not just the goals of acomputer tutor, but they are the goals of a human tutor as well. Alltutors must look at a student's work to help identify and correct errorsas well as learn the material. One of the most difficult tasks facing atutor is the difficult task of balancing the appropriate amount ofassistance provided the student to complete the task with therequirement to help the student learn the material.

Model of Feedback

A preferred embodiment utilizes feedback to address the balancing task.The theory is centered on the idea of severity. Severe errors requiresevere feedback while mild errors require mild feedback. If a studentwrites a paper on the wrong subject, a human tutor will spend littletime reviewing the paper, but instead, identify it as a serious mistakeand ask the student to rewrite the paper. If the student simply missesone paragraph of the argument, then the tutor will focus the student onthat paragraph. Finally, if the paper is correct except for a couple ofspelling mistakes, the tutor will point out the specific mistakes andask the student to correct them. The point is that because a tutor and astudent do not want to waste each others' time, they will match theseverity of the error with the severity of the feedback.

In the ICAT model of feedback, there are four levels of severity oferror and four corresponding levels of feedback. The tutor goes throughthe student's work, identifies the severity of the error and thenprovides the corresponding level of feedback.

Educational Categories of Feedback ERROR FEEDBACK Error Feedback TypeDescription Type Description 1. None No errors exist. 1. PraiseConfirmation that the The student's student completed the work isperfect. task correctly. Example: Great. You have journalized allaccounts correctly. I am happy to see you recognized we are pay- ing formost of our bills “on account”. 2. Syntactic There may be 2. PolishTells the student the spelling specific actions he did mistakes orincorrectly, and other syntactic possibly correct them errors. As a forhim. designer, you Example: should be There are one or two confidentthat errors in your work. It the student will looks like you mis- havemastered classified the purchase the material at of the fax as a cashthis point. purchase when it is really a purchase on account. 3. Local Aparagraph of 3. Focus Focus the student on a paper is this area of hiswork. missing or the Point out that he does student has not understandat least made a number one major concept. of mistakes all Example: inone area. Looking over your The student work, I see that you do clearlydoes not not understand the understand this area. concept of “onaccount”. Why don't you review that concept and review your work forerrors. 4. Global The student has 4. Redirect Restate the goal of thewritten on the activity and tell the wrong subject student to reviewmain or there are concepts and retry the mistakes all activity. over theExample: student's work There are lots of which indicates mistakesthroughout he does not your work. You need understand most to thinkabout what of the concepts type of transaction in the activity. eachsource document represents before journalizing it.

Returning to the analogy of helping someone write a paper, if thestudent writes on the wrong subject, this as a global error requiringredirect feedback. If the student returns with the paper rewritten, butwith many errors in one area of the paper, focus feedback is needed.With all of those errors fixed and only spelling mistakes—syntacticmistakes—polish feedback is needed. When all syntactic mistakes werecorrected, the tutor would return praise and restate why the student hadwritten the correct paper.

Focusing on the educational components of completing a task is notenough. As any teacher knows, student will often try and cheat their waythrough a task. Students may do no work and hope the teacher does notnotice or the student may only do minor changes in hope of a hint orpart of the answer. To accommodate these administrative functions, thereare three additional administrative categories of feedback.

Administrative Categories of Feedback Error Description FeedbackDescription No work The student has made Master- Tell the student thathe done since no changes since the mind has done no work and last reviewlast time he asked for that a substantial the tutor to review his amountof work needs work. to be completed before review. Example: You havedone no work since I last reviewed your work. Please try and correct atleast three journal entries before asking me to review your work again.All work If a designer wants to Incom- State that the student is notgive an interim report plete- has not completed all complete of how thestudent is continue of the work required, but a doing before every- butyou will review substantial thing is done, they what the student hasamount of he would use done so far. work has incomplete--continue.Example: been done It looks like you have not finished journal- izing,but I will review what you have done up to this point. The first threeentries are correct. All work If a user has not Incom- State thatnothing has is not completed enough plete- been attempted and completework to receive stop point to the first and a feedback, this categoryaction to be taken. substantial is used. Example: amount of It lookslike you have work is not done no work journal- complete izing. Whydon't you start by trying to journalize the fax purchase.

The administrative and the educational categories of feedback accountfor every piece of feedback a designer can write and a student canreceive. To provide a better understanding of how the feedback workstogether, an example is provided below.

Feedback Example

The following example is a GBS training application in which new financeprofessionals are taught the fundamentals of finance management. Astudent has a toolbar to navigate and also to access some of theapplication-level features of the application. The toolbar is theL-shaped object across the top and left of the interface. The topsection of the toolbar allows the user to navigate to tasks within thecurrent Activity. The left section of the toolbar allows the student toaccess other features of the application, including feedback. Thestudent can have his deliverables analyzed and receive feedback byclicking on a team button.

In this task, the student must journalize twenty-two invoices and othersource documents to record the flow of budget dollars between internalaccounts. (Note: “Journalizing”, or “Journalization”, is the process ofrecording journal entries in a general ledger from invoices or othersource documents during an accounting period. The process entailscreating debit and balancing credit entries for each document. At thecompletion of this process, the general ledger records are used tocreate a trial balance and subsequent financial reports.) The studenthas several controls on the screen that must be manipulated to completethe task. The upper left area of the screen shows the currenttransaction. Each transaction has a corresponding journal entry. Thebottom of the screen shows the current journal entry. The Top two linesof the journal entry are for Debits (DR) and the bottom two lines arefor Credits (CR). As the student uses the ‘Back’ and ‘Next’ buttons topage through the transactions, the journal entry is also paged to stayin sync.

FIG. 12 is a GBS display in accordance with a preferred embodiment. Theupper right area of the screen shows the account list. There are fourtypes of accounts: Assets, Liabilities & Equity, Revenues, and Expenses.The user clicks on one of the tabs to show the accounts of thecorresponding type. The student journalizes a transaction by dragging anaccount from the account list onto the journal entry Debits or Credits.The student then enters the dollar amounts to debit or credit eachaccount in the entry. In the interface, as in real life, the student canhave multi-legged journal entries (i.e., debiting or crediting multipleaccounts).

A Toolbar 1200 and the first transaction of this Task 1210 appearprominently on the display. The student can move forward and backthrough the stack of transactions. For each transaction, the studentmust identify which accounts to debit and which to credit. When thestudent is done, he clicks the Team button.

FIG. 13 is a feedback display in accordance with a preferred embodiment.The student may attempt to outsmart the system by submitting withoutdoing anything. The ICAT system identifies that the student has not donea substantial amount of work and returns the administrative feedbackdepicted in FIG. 13. The feedback points out that nothing has been done,but it also states that if the student does some work, the tutor willfocus on the first few journal entries.

FIG. 14 illustrates a display in which a student has made some mistakesin accordance with a preferred embodiment. The student tries tojournalize the transaction depicted in FIG. 14 which reflects thecapital needed to start the business. The student attempts to journalizethe transaction by debiting the paid-in capital account and creditingthe cash account for $210,000. Similarly, the student attempts tojournalize the purchase of Government Bonds by debiting accountsreceivable and crediting cash for $150,000 as shown in FIG. 15. FIG. 15illustrates a journal entry simulation in accordance with a preferredembodiment.

FIG. 16 illustrates a simulated Bell Phone Bill journal entry inaccordance with a preferred embodiment. The journal entry isaccomplished by debiting Utilities Expenses and Crediting Cash for $700each.

FIG. 17 illustrates a feedback display in accordance with a preferredembodiment. After attempting to journalize the first three transactions,the student submits his work and receives the feedback depicted in FIG.17. The feedback starts by focusing the student on the area of workbeing evaluated. The ICAT states that it is only looking at the firstthree journal entries. The feedback states that the first two entriesare completely wrong, but the third is close. If the student had madelarge mistakes on each of the first three transactions, then the ICATmay have given redirect feedback, thinking a global error occurred. Thethird bullet point also highlights how specific the feedback can become,identifying near misses.

FIGS. 18 and 19 illustrate a feedback display in accordance with apreferred embodiment. As a student attempts to correct transactions oneand two unsuccessfully, the tutor starts to provide hints, stating thatthe student should debit an asset account and credit an equity account.The ICAT continues to focus on the errors in the first three sourcedocuments and is giving progressively more specific hints.

FIG. 20 illustrates a feedback display in accordance with a preferredembodiment. With the specific hints provided as illustrated in FIG. 19,the student correctly journalizes the source document. The ICAT,however, continues to focus the student on these first three journalentries as illustrated in FIG. 20. The student finally completes thefirst three entries correctly. The feedback illustrated in FIG. 20informs the student of his success and instructs him to try to completethe rest of the transaction before submitting his deliverable again.This example illustrates the use of an effective technique called“baby-stepping”. The student is helped through a small portion of thework to get him introduced to the concepts and the interface.

After completing this, he is forced to attempt all of the remaining workbefore getting substantive feedback. This technique can be used to mimicthe kind of interactions one could expect to receive from a mentor inreal life. The three transactions above show a tiny fraction of thedepth of student analysis and richness of remediation that the ICAT iscapable of delivering.

As mentioned earlier in the Remediation Model section, the tutor playstwo roles in any course. First, the tutor reviews the student's work andhelps him/her understand the task and the associate concepts. Second thetutor is gatekeeper between sections. The tutor will not allow studentsto proceed to the next section of the course until they have gotten thecurrent section correct. To monitor student progress, the course hasbeen broken into two components:

Activity

An activity is a business event, such as planning a company's financialsor closing the books. Business events set the context of the course.Students learn the content so that they can complete the goalsassociates with each business event. The power of a GBS is in how itembeds the content a student needs to learn within the context of thebusiness events.

Task

A task is a business deliverable that must be completed as part of abusiness event. Example tasks include completing journal entries whileclosing the books. There may be many Tasks in an activity, just as theremay be many deliverables required to react to a business event in thereal world. Deliverables produced in this application include abreak-even analysis, a transaction journal, a cost report, and a ratioanalysis. The role of the tutor is to help the students complete thebusiness deliverables associated with any business event. Students canalways go backward, but they are limited from going forward, until theICAT says that the business deliverable meets the requiredspecifications. It is useful to think of the ICAT as a boss who reviewsyour work. The boss will not let you go on to the next task, or businessdeliverable, until you have correctly completed the current task. Tohelp explain the concepts of an activity and task, here is a descriptionof an ICAT implementation in accordance with a preferred embodiment.

A training application utilizing ICAT for a large product company ispresented as an example. The training application is a revision of thefirst semester of a two year financial training program. Students learnfinance by managing a simulated bicycle company for three years andusing finance to solve business problems. At four places in the course,the students come together to present their analyses of the business.These presentations are live presentations to real business executives.

In preparation for the pitches, the students complete computer-basedmodules. There are two major sections to each module, the accountingconcepts and the activities. Students learn the concepts and ideas inthe accounting concepts and apply the concepts in the activities. All ofthe modules together represent the different phases associated withrunning a business: Start Operations, Analyze Operations and ImproveOperations. Each computer-based activity represents a business event,such as closing the books of the company. These business events providecontext for the content the students learn in the course. In this way,students not only learn what the concepts are but when, how and why theyshould use them.

Business Events-Activities 1. Financial Planning 2. RecordingTransactions 3. Recording Transactions 4. Closing the Books 5. Analyzethe Books 6. Improve Operations

FIG. 21 illustrates a simulation display in accordance with a preferredembodiment. To show how the business events impact the company on a dayto day basis, students complete a set of deliverables associated witheach business event. The business deliverables students create in thetraining application are varied in form and content. Some examplebusiness deliverables are listed below in accordance with a preferredembodiment.

1. An Analysis of Proforma Financial Statements

Students perform break-even analysis to determine which of twelvebusiness strategies to pursue.

2. Journal Entries

Student journalize 20 of the transactions which occur in the third yearof operations.

3. Summaries of Interviews With Employees About Operating Plan Variances

Students get behind the numbers and figure out what is driving thevariances.

Design Scenario

This Scenario illustrates how the tools are used to support conceptualand detailed design of a BusSim application. FIG. 22 illustrates thesteps of the first scenario in accordance with a preferred embodiment.The designer has gathered requirements and determined that to supportthe client's learning objectives, a task is required that teachesjournalization skills. The designer begins the design first by learningabout journalization herself, and then by using the Knowledge Workbenchto sketch a hierarchy of the concepts she want the student to learn. Atthe most general level, she creates a root concept of ‘Journalization’.She refines this by defining sub-concepts of ‘Cash relatedtransactions’, ‘Expense related Transactions’, and ‘Expense on accounttransactions’. These are each further refined to whatever level of depthis required to support the quality of the learning and the fidelity ofthe simulation.

The designer then designs the journalization interface. Since a greatway to learn is by doing, she decides that the student should be askedto Journalize a set of transactions. She comes up with a set oftwenty-two documents that typify those a finance professional might seeon the job. They include the gamut of Asset, Expense, Liability andEquity, and Revenue transactions. Also included are some documents thatare not supposed to be entered in the journal. These ‘Distracters’ areincluded because sometimes errant documents occur in real life. Thedesigner then uses the Domain Model features in the Knowledge Workbenchto paint a Journal. An entity is created in the Domain Model torepresent each transaction and each source document. Based on thetwenty-two documents that the designer chose, she can anticipate errorsthat the student might make. For these errors, she creates topics offeedback and populates them with text. She also creates topics offeedback to tell the student when they have succeeded. Feedback Topicsare created to handle a variety of situations that the student maycause. The next step is to create profiles that the will trigger thetopics in the concept tree (this task is not computational in nature, sothe Transformation Component does not need to be configured). A profileresolves to true when its conditions are met by the student's work. Eachprofile that resolves to true triggers a topic.

To do some preliminary testing on the design, the designer invokes theStudent Simulator Test Workbench. The designer can manipulate the DomainModel as if she were the student working in the interface. She dragsaccounts around to different transactions, indicating how she would likethem journalized. She also enters the dollar amounts that she would liketo debit or credit each account. She submits her actions to thecomponent engines to see the feedback the student would get if he hadperformed the activity in the same way. All of this occurs in the testbench without an application interface.

The last step in this phase is low-fi user testing. A test studentinteracts with a PowerPoint slide or bitmap of the proposed applicationinterface for the Journalization Task. A facilitator mimics his actionsin the test bench and tells him what the feedback would be. Thissimplifies low-fi user testing and helps the designer to identifyusability issues earlier in the design when they are much cheaper toresolve.

Build Scenario

FIGS. 23 and 24 illustrate the steps associated with a build scenario inaccordance with a preferred embodiment. The instructional designercompletes the initial interaction and interface designs as seen in theprevious Scenario. After low-fi user testing, the Build Phase begins.Graphic artists use the designs to create the bitmaps that will make upthe interface. These include bitmaps for the buttons, tabs, andtransactions, as well as all the other screen widgets. The developerbuilds the interface using the bitmaps and adds the functionality thatnotifies the Domain Model of student actions. Standard event-drivenprogramming techniques are used to create code that will react to eventsin the interface during application execution and pass the appropriateinformation to the Domain Model. The developer does not need to have anydeep knowledge about the content because she does not have to build anylogic to support analysis of the student actions or feedback. Thedeveloper also codes the logic to rebuild the interface based on changesto the domain model.

A few passes through these steps will typically be required to get theapplication communicating correctly with the components. The debugutilities and Regression Test Workbench streamline the process. Afterthe application interface and component communication are functioning asdesigned, the task is migrated to Usability testing.

Test Scenario

This scenario demonstrates the cycle that the team goes through to testthe application. It specifically addresses usability testing, but it iseasy to see how the tools also benefit functional and cognition testing.Again, we will use the Journalization Task as an example. FIG. 24illustrates a test scenario in accordance with a preferred embodiment.The test students work through the journalization activity. One of thestudents has made it over half way through the task and has justattempted to journalize the sixteenth transaction. The student submitsto the Financial Coach, but the feedback comes back blank. The studentnotifies the facilitator who right-clicks on the Financial Coach's facein the feedback window. A dialog pops up that shows this is thetwenty-seventh submission and shows some other details about thesubmission. The facilitator (or even the student in recent efforts)enters a text description of the problem, and fills out some otherfields to indicate the nature and severity of the problem. All thestudent's work and the feedback they got for the twenty-sevensubmissions is posted to the User Acceptance Test (UAT) archivedatabase.

The instructional designer can review all the student histories in theUAT database and retrieve the session where the student in questionattempted the Journalization Task. The designer then recreates theproblem by replaying the student's twenty-seven submissions through thecomponent engines using the Regression Test Workbench. The designer canthen browse through each submission that the student made and view thework that the student did on the a submission, the feedback the studentgot, and the facilitator comments, if any. Now the designer can use thedebugging tools to determine the source of the problem. In a fewminutes, she is able to determine that additional profiles and topicsare needed to address the specific combinations of mistakes the studentmade. She uses the Knowledge Workbench to design the new profiles andtopics. She also adds a placeholder and a script for a video war storythat supports the learning under these circumstances. The designer savesthe new design of the task and reruns the Regression Test Workbench onthe student's session with the new task design. After she is satisfiedthat the new profiles, topics, and war stories are giving the desiredcoverage, she ships the new task design file to user testing and it'srolled out to all of the users.

This example illustrates how a high effort, uncertain process (that oncetook days) can be reduced to a few hours using the BusSim Toolset. Cycletime can be reduced dramatically, and complexity, risk and difficultycan be almost eliminated. It shows the sharp contrast with thetraditional development approach where new designs and new code can havemany unintended consequences that are difficult to test.

Execution Scenario: Student Administration

FIG. 25 illustrates how the tool suite supports student administrationin accordance with a preferred embodiment. When a student first enters acourse she performs a pre-test of his financial skills and fills out aninformation sheet about his job role, level, etc. This information isreported to the Domain Model. The Profiling Component analyzes thepre-test, information sheet, and any other data to determine thespecific learning needs of this student. A curriculum is dynamicallyconfigured from the Task Library for this student. The applicationconfigures its main navigational interface (if the app has one) toindicate that this student needs to learn Journalization, among otherthings.

As the student progresses through the course, his performance indicatesthat his proficiency is growing more rapidly in some areas than inothers. Based on this finding, his curriculum is altered to give himadditional Tasks that will help him master the content he is havingtrouble with. Also, Tasks may be removed where he has demonstratedproficiency. While the student is performing the work in the Tasks,every action he takes, the feedback he gets, and any other indicators ofperformance are tracked in the Student Tracking Database. Periodically,part or all of the tracked data are transmitted to a central location.The data can be used to verify that the student completed all of thework, and it can be further analyzed to measure his degree of mastery ofthe content.

Execution Scenario: Student Interaction

FIG. 26 illustrates a suite to support a student interaction inaccordance with a preferred embodiment. In this task the student istrying to journalize invoices. He sees a chart of accounts, an invoice,and the journal entry for each invoice. He journalizes a transaction bydragging and dropping an account from the chart of accounts onto the‘Debits’ or the ‘Credits’ line of the journal entry and entering thedollar amount of the debit or credit. He does this for each transaction.

As the student interacts with the interface, all actions are reported toand recorded in the Domain Model. The Domain Model has a meta-modeldescribing a transaction, its data, and what information a journal entrycontains. The actions of the student populates the entities in thedomain model with the appropriate information. When the student isready, he submits the work to a simulated team member for review. Thissubmission triggers the Analysis-Interpretation cycle. TheTransformation Component is invoked and performs additional calculationson the data in the Domain Model, perhaps determining that Debits andCredits are unbalanced for a given journal entry.

The Profiling Component can then perform rule-based pattern matching onthe Domain Model, examining both the student actions and results of anyTransformation Component analysis. Some of the profiles fire as theyidentify the mistakes and correct answers the student has given. Anyprofiles that fire activate topics in the Remediation Component. Afterthe Profiling Component completes, the Remediation Component is invoked.The remediation algorithm searches the active topics in the tree ofconcepts to determine the best set of topics to deliver. This set maycontain text, video, audio, URLs, even actions that manipulate theDomain Model. It is then assembled into prose-like paragraphs of textand media and presented to the student. The text feedback helps thestudent localize his journalization errors and understand why they arewrong and what is needed to correct the mistakes. The student ispresented with the opportunity to view a video war story about the taxand legal consequences that arise from incorrect journalization. He isalso presented with links to the reference materials that describe thefundamentals of journalization.

The Analysis-Interpretation cycle ends when any coach items that resultin updates to the Domain Model have been posted and the interface isredrawn to represent the new domain data. In this case, the designerchose to highlight with a red check the transactions that the studentjournalized incorrectly.

III. The Functional Definition of the ICAT

This section describes the feedback processes in accordance with apreferred embodiment. For each process, there is a definition of theprocess and a high-level description of the knowledge model. Thisdefinition is intended to give the reader a baseline understanding ofsome of the key components/objects in the model, so that he can proceedwith the remaining sections of this paper. Refer to the DetailedComponents of the ICAT for a more detailed description of each of thecomponents within each knowledge model. To gain a general understandingof the ICAT, read only the general descriptions. To understand the ICATdeeply, read this section and the detailed component section regardingknowledge models and algorithms. These processes and algorithms embodythe feedback model in the ICAT. There are six main processes in theICAT, described below and in more detail on the following pages.

Remediation Process Diagram

FIG. 27 illustrates the remediation process in accordance with apreferred embodiment. Remediation starts as students interact with theapplication's interface (process #1). As the student tries to completethe business deliverable, the application sends messages to the ICATabout each action taken (process #2). When the student is done andsubmits work for review, the ICAT compares how the student completed theactivity with how the designer stated the activity should be completed(this is called domain knowledge). From this comparison, the ICAT get acount of how many items are right, wrong or irrelevant process #3). Withthe count complete, the ICAT tries to fire all rules (process #4). Anyrules which fire activate a coach topic (process #5). The feedbackalgorithm selects pieces of feedback to show and composes them intocoherent paragraphs of text (process #6). Finally, as part of creatingfeedback text paragraphs, the ICAT replaces all variables in thefeedback with specifics from the student's work. This gives the feedbackeven more specificity, so that it is truly customized to each student'sactions.

1. Student Interacts With Interface to Create Business Deliverable

Description

The student completes the deliverables of the Task by interacting withthe interface objects. These actions may be button clicks, dragging oftext, selection of items from a list, etc. An example is theJournalization task shown below. FIG. 28 illustrates a display ofjournalization transactions in accordance with a preferred embodiment.To interact with the display, the student must journalize thetwenty-four transactions presented. To journalize a transaction, thestudent clicks the “next” and “previous” buttons to move betweentransactions. Once at a transaction, the student clicks and drags anaccount name from the chart of accounts—which is split into Assets,Liabilities, Revenues and Expenses—onto the debit or credit side of thejournal entry. Once the journal entry has been made, the student musttype in how much to debit or credit. Each one of these buttons,draggable items, and text fields are interface objects which can bemanipulated.

Knowledge Model

Interface Objects

In any GBS Task, the student must manipulate controls on the applicationinterface to complete the required deliverables. FIG. 29 illustrates theobjects for the journalization task in accordance with a preferredembodiment.

The following abstract objects are used to model all the various typesof interface interactions.

SourceItem

A SourceItem is an object the student uses to complete a task. In thejournalization example, the student makes a debit and credit for eachtransaction. The student has a finite set of accounts with which torespond for each transaction. Each account that appears in the interfacehas a corresponding SourceItem object. In other words, the items thestudent can manipulate to complete the task (account names) are calledSourceItems.

Source

A Source is an object that groups a set of SourceItem objects together.Source objects have a One-To-Many relationship with SourceItem objects.In the journalization example, there are four types of accounts: Assets,Liabilities and Equity, Revenues, and Expenses. Each Account is of oneand only one of these types and thus appears only under the appropriatetab. For each of the Account type tabs, there is a corresponding SourceObject.

Target

A Target is a fixed place where students place SourceItems to complete atask. In the journalization example, the student places accounts on twopossible targets: debits and credits. The top two lines of the journalentry control are Debit targets and the bottom two lines are Credittargets. These two targets are specific to the twelfth transaction.

TargetPage

A TargetPage is an object that groups a set of Target objects together.TargetPage objects have a One-To-Many relationship with Target objectsOust like the Source to SourceItem relationship). In the journalizationexample, there is one journal entry for each of the twenty-twotransactions. For each journal entry there is a corresponding TargetPageobject that contains the Debits Target and Credits Target for thatjournal entry.

2. Reporting Student Actions to the ICAT

Description

As the student manipulates the application interface, each action isreported to the ICAT. In order to tell the ICAT what actions were taken,the application calls to a database and asks for a specific interfacecontrol's ID. When the application has the ID of the target control andthe SourceItem control, the application notifies the ICAT about theTarget to SourceItem mapping. In other words, every time a studentmanipulates a source item and associates it with a target (e.g.,dragging an account name to a debit line in the journal), the useraction is recorded as a mapping of the source item to the target. Thismapping is called a UserSourceItemTarget. FIG. 30 illustrates themapping of a source item to a target item in accordance with a preferredembodiment.

3. Student Submits Deliverables to One Team Member

Description

When the student is ready, he submits his work to one of the simulatedteam members by clicking on the team member's icon. When the ICATreceives the student's work, it calculates how much of the work iscorrect by concept Concepts in our journalization activity will includeDebits, Credits, Asset Accounts, etc. For each of these concepts, theICAT will review all student actions and determine how many of thestudent actions were correct. In order for the ICAT to understand whichtargets on the interface are associated with each concept, the targetsare bundled into target groups and prioritized in a hierarchy. FIG. 31illustrates target group bundles in accordance with a preferredembodiment. For each target group—or concept, such as debit—a number ofaggregate values will be calculated. These aggregate values determinehow many student actions were right, wrong or irrelevant.

Knowledge Model

TargetGroup

A TargetGroup object represents a concept being learned. It is a groupof Target objects related on a conceptual level. The TargetGroup objectsin a Task are arranged in a hierarchy related to the hierarchy ofconcepts the student must learn. By analyzing the student's responses tothe Targets in a TargetGroup, the ICAT can determine how well a studentknows the concept. By utilizing the conceptual hierarchy of TargetGroupsthe ICAT can determine the most appropriate remediation to deliver tohelp the student understand the concepts.

TargetGroup Hierarchy

The TargetGroup objects in a Task are arranged in a hierarchical treestructure to model the varying specificity of concepts and sub-conceptsbeing learned in the Task. The designer defines the parent-childrelationships between the TargetGroups to mimic the relationships of thereal world concepts. This hierarchy is used in the determination of themost appropriate feedback to deliver. Concepts that are higher (moreparent-like) in the TargetGroup structure are remediated before conceptsthat are modeled lower (children, grandchildren, etc.) in the tree. FIG.32 illustrates a TargetGroup Hierarchy in accordance with a preferredembodiment.

In the journalization example, the main concept being taught isjournalization. The concept of journalization can be divided into morespecific sub-concepts, for example journalizing cash-for-expensetransactions and journalizing expense-on-account transactions. These mayfurther be divided as necessary. The designer teaches this conceptualknowledge to the ICAT by creating a TargetGroup called “JournalizingTransactions” with two child TargetGroups “Journalizing Cash for ExpenseTransactions” and “Journalizing Expense On Account Transactions”. Thetop-most TargetGroup in the Task, “Journalizing Transactions” containsall of the transactions in the Task. Child target groups will includejust the first three transactions and transactions four to twenty.

Therefore when the when the ICAT determines how much of the task iscorrect, it will calculate values for the first three journal entriesand the next sixteen. Calculating these two separate numbers will allowthe ICAT to provide specific feedback about the first three and separatefeedback about the next sixteen transactions. Here is a section of thetarget group hierarchy for the journalize task. FIG. 33 illustrates asmall section the amount of feedback in accordance with a preferredembodiment. By analyzing the responses to the targets in the each of thetargetgroups, we can determine how many of the transactions the studenthas attempted, whether mistakes were made, what the mistakes were, etc.We can then assemble feedback that is very specific to the way thestudent completed the deliverables. By analyzing the student's responsesto a group of conceptually related requests, we can determine the degreeof success with which the student is learning the concept.

4. ICAT Analyzes Deliverables With Rules

Description

After the ICAT has calculated the aggregate values for the student'sdeliverables, it analyzes the deliverables by attempting to fire all ofthe Rules for that task. Rules that can fire, activate CoachTopics. FIG.34 illustrates an analysis of rules in accordance with a preferredembodiment.

5. Select Appropriate Remediation Coach Topics

Description

Once all possible coach topics are activated, a feedback selectionanalyzes the active pieces of remediation within the concept hierarchyand selects the most appropriate for delivery. The selected pieces offeedback are then assembled into a cohesive paragraph of feedback anddelivered to the student. FIG. 35 illustrates a feedback selection inaccordance with a preferred embodiment.

Feedback Selection Algorithm

After the ICAT has activated CoachTopics via Rule firings, the FeedbackSelection Algorithm is used to determine the most appropriate set ofCoachItems (specific pieces of feedback text associated with aCoachTopic) to deliver. The Algorithm accomplishes this by analyzing theconcept hierarchy (TargetGroup tree), the active CoachTopics, and theusage history of the CoachItems. FIG. 36 is a flowchart of the feedbacklogic in accordance with a preferred embodiment. There are five mainareas to the feedback logic which execute sequentially as listed below.First, the algorithm looks through the target groups and looks for thetop-most target group with an active coach topic in it. Second, thealgorithm then looks to see if that top-most coach item is praisefeedback. If it is praise feedback, then the student has correctlycompleted the business deliverable and the ICAT will stop and returnthat coach item. Third, if the feedback is not Praise, then the ICATwill look to see if it is redirect, polish, mastermind orincomplete-stop. If it is any of these, then the algorithm will stop andreturn that feedback to the user. Fourth, if the feedback is focus, thenthe algorithm looks to the children target groups and groups any activefeedback in these target groups with the focus group header. Fifth, oncethe feedback has been gathered, then the substitution language is runwhich replaces substitution variables with the proper names.

Once the ICAT has chosen the pieces of feedback to return, the feedbackpieces are assembled into a paragraph. With the paragraph assembled, theICAT goes through and replaces all variables. There are specificvariables for SourceItems and Targets. Variables give feedbackspecificity. The feedback can point out which wrong SourceItems wereplaced on which Targets. It also provides hints by providing one or twoSourceItems which are mapped to the Target.

IV. The ICAT Development Methodology for Creating Feedback

The Steps Involved in Creating Feedback

The goal of feedback is to help a student complete a businessdeliverable. The tutor needs to identify which concepts the studentunderstands and which he does not. The tutor needs to tell the studentabout his problems and help him understand the concepts.

There are seven major steps involved in developing feedback for anapplication.

First, creating a strategy—The designer defines what the student shouldknow. Second, limit errors through interface—The designer determines ifthe interface will identify some low level mistakes. Third, creating atarget group hierarchy—The designer represents that knowledge in thetutor. Fourth, sequencing the target group hierarchy—The designer tellsthe tutor which concepts should be diagnosed first. Fifth, writingfeedback—The designer writes feedback which tells the student how he didand what to do next. Sixth, writing Levels of Feedback—The designerwrites different levels of feedback in case the student makes the samemistake more than once. Seventh, writing rules—The designer definespatterns which fire the feedback.

Creating a Feedback Strategy

A feedback strategy is a loose set of questions which guide the designeras he creates rules and feedback. The strategy describes what thestudent should learn, how he will try and create the businessdeliverable and how an expert completes the deliverable. The goal of theapplication should be for the student to transition from the novicemodel to the expert model.

What should the student know after using the application?

The first task a designer needs to complete is to define exactly whatknowledge a student must learn by the end of the interaction. Should thestudent know specific pieces of knowledge, such as formulas? Or, shouldthe student understand high level strategies and detailed businessprocesses? This knowledge is the foundation of the feedback strategy.The tutor needs to identify if the student has used the knowledgecorrectly, or if there were mistakes. An example is the journal task.For this activity, students need to know the purpose of the journalizingactivity, the specific accounts to debit/credit, and how much todebit/credit. A student's debit/credit is not correct or incorrect inisolation, but correct and incorrect in connection with the dollarsdebited/credited.

Because there are two different types of knowledge—accounts todebit/credit and amounts to debit/credit—the feedback needs to identifyand provide appropriate feedback for both types of mistakes.

How will a novice try and complete the task?

Designers should start by defining how they believe a novice will tryand complete the task. Which areas are hard and which are easy for thestudent. This novice view is the mental model a student will bring tothe task and the feedback should help the student move to an expertview. Designers should pay special attention to characteristic mistakesthey believe the student will make. Designers will want to createspecific feedback for these mistakes. An example is mixing up expenseaccounts in the journal activity. Because students may mix up some ofthese accounts, the designer may need to write special feedback to helpclear up any confusion.

How does an expert complete the task?

This is the expert model of completing the task. The feedback shouldhelp students transition to this understanding of the domain. Whencreating feedback, a designer should incorporate key features of theexpert model into the praise feedback he writes. When a studentcompletes portion of the task, positive reinforcement should be providedwhich confirms to the student that he is doing the task correctly andcan use the same process to complete the other tasks.

These four questions are not an outline for creating feedback, but theydefine what the feedback and the whole application needs to accomplish.The designer should make sure that the feedback evaluates all of theknowledge a student should learn. In addition, the feedback should beable to remediate any characteristic mistakes the designer feels thestudent will make. Finally, the designer should group feedback so thatit returns feedback as if it were an expert. With these componentsidentified, a designer is ready to start creating target grouphierarchies.

Limit Errors Through Interface

When the designer defines a feedback strategy, the designer defines theskills he wants the student to learn and the mistakes he thinks thestudent will make. Not all of the mistakes need to be corrected withICAT generated feedback, some can be limited with or remediated throughthe interface. Limiting mistakes with the interface simply means thatthe system pops-up a message as the student works, identifying amistake. An example interface corrected error is in the journalizationactivity when the interface points out that debits do not equal credits.Here, this is a low level mistake which is more appropriate to remediatethrough the interface than through the ICAT. The application simplycheck to see if the debit number equal the credit number and if they donot then the system message is returned. FIG. 37 illustrates an exampleof separating out some mistakes for the interface to catch and othersfor the ICAT to catch has positive and negative impacts in accordancewith a preferred embodiment.

Positive

The most obvious reason for eliminating mistakes through the interfaceis that can be easier for the designer and developer to catch them atthis level than to leave them for the ICAT.

Negative

The reason to avoid interface-driven feedback is that it splinters thefeedback approach which can make the job of creating a coherent feedbackapproach more difficult.

Because there are positive and negative repercussions, designers need toselect the when to remediate through the interface carefully. Thecriteria for making the decision is if the mistake is a low level dataentry mistake or a high level intellectual mistake. If the mistake is alow level mistake, such as miss-typing data, it may be appropriate toremediate via the interface. If the designer decides to have theinterface point out the mistakes, it should look as if the systemgenerated the message. System generated messages are mechanical checks,requiring no complex reasoning. In contrast, complex reasoning, such aswhy a student chose a certain type of account to credit or debit shouldbe remediated through the ICAT.

System Messages

It is very important that the student know what type of remediation heis going to get from each source of information. Interface basedremediation should look and feel like system messages. They should use adifferent interface from the ICAT remediation and should have adifferent feel. In the journalization task described throughout thispaper, there is a system message which states “Credits do not equaldebits.” This message is delivered through a different interface and theblunt short sentence is unlike all other remediation.

The motivation for this is that low level data entry mistakes do notshow misunderstanding but instead sloppy work. Sloppy-work mistakes donot require a great deal of reasoning about why they occurred instead,they simply need to be identified. High-level reasoning mistakes,however, do require a great deal of reasoning about why they occurred,and the ICAT provides tools, such as target groups, to help with complexreasoning. Target group hierarchies allow designers to group mistakesand concepts together and ensure that they are remediated at the mostappropriate time (i.e., Hard concepts will be remediated before easyconcepts). Timing and other types of human-like remediation require theICAT; other low-level mistakes which do not require much-reasoninginclude:

Incomplete

If the task requires a number of inputs, the interface can check thatthey have all been entered before allowing the student to proceed. Bycatching empty fields early in the process, the student may be saved thefrustration of having to look through each entry to try and find theempty one.

Empty

A simple check for the system is to look and see if anything has beenselected or entered. If nothing has been selected, it may be appropriatefor the system to generate a message stating “You must complete X beforeproceeding”.

Numbers not Matching

Another quick check is matching numbers. As in the journalizationactivity, is often useful to put a quick interface check in place tomake sure numbers which must match do. Small data entry mistakes areoften better remediated at the interface level than at the tutor orcoach level (when they are not critical to the learning objectives ofthe course).

There are two main issues which must be remembered when using theinterface to remediate errors. First, make sure the interface isremediating low level data entry errors. Second, make sure the feedbacklooks and feels different from the ICAT feedback. The interface feedbackshould look and feel like it is generated from the system while the ICATfeedback must look as if it were generated from an intelligent coach whois watching over the student as he works.

Creating the Target Group Hierarchy

Target groups are sets of targets which are evaluated as one. Returningto the severity principle of the feedback theory, it is clear that thetutor needs to identify how much of the activity the student does notunderstand. Is it a global problem and the student does not understandanything about the activity? Or, is it a local problem and the studentsimply is confused over one concept? Using the feedback algorithmdescribed earlier, the tutor will return the highest target group inwhich there is feedback. This algorithm requires that the designer startwith large target groups and make sub-groups which are children of thelarger groups. The ICAT allows students to group targets in more thanone category. Therefore a debit target for transaction thirteen can bein a target group for transaction thirteen entries as well as a targetgroup about debits and a target group which includes all sourcedocuments. Target should be grouped with four key ideas in mind. Targetgroups are grouped according to:

Concepts taught

Interface constraints

Avoidance of information overload

Positive reinforcement

The most important issue when creating target groups is to create themalong the concepts students need to know to achieve the goal. Groupingtargets into groups which are analogous to the concepts a student needsto know, allows the tutor to review the concepts and see which conceptsconfuse the student. As a first step, a designer should identify in anunstructured manner all of the concepts in the domain. This first passwill be a large list which includes concepts at a variety ofgranularities, from small specific concepts to broad general concepts.These concepts are most likely directly related to the learningobjectives of the course.

With all of the concepts defined, designers need to identify all of thetargets which are in each target group. Some targets will be in morethan one target group. When a target is in more than one target group,it means that there is some type of relationship such as a childrelationship or a part to whole relationship. The point is not to createa structured list of concepts but a comprehensive list. Structuring theminto a hierarchy will be the second step of the process.

In the journalization activity, the largest concept is the recording atransaction. Other important ideas are debits and credits. Debit andcredit targets, however, are included in the overall transaction targetgroup which means that it is either a part-whole relationship or a childrelationship. FIG. 38 is a block diagram of the hierarchicalrelationship of a transaction in accordance with a preferred embodiment.

Concepts Taught: Part-whole Concepts

With all of the target groups laid out, the designer needs to identifythe relationships between concepts. One type of relationship is thepart-whole relationship. Part-whole relationships—as the namedenotes—identified which sub-components make up larger concepts.Identifying these relationships is important because the tutor will wantto see if the student does not understand the whole concept or just onepart. If there are no major errors in the concept as a whole, then thetutor will look to see if the student made any major errors in one partof the concept.

Example:

In the journalizing activity, there will be a target group calledtransaction. In transaction, there are two parts: debits and credits.When the tutor reviews the student's work, if there are no problems withthe target group transactions, then the tutor will go to the next leveland look for errors in the target group debits and credits. Becausedebits and credits are included in an overall transaction, there is apart-whole relationship to the concept transaction.

Concept Taught: Child Concepts

In addition to part-whole relationships, designers need to identifychild-parent relationships. In contrast to part-whole relationships,child-parent relationships define instances of abstract concepts. Anexample is “The dictionary is a book”. “Dictionary” is a child conceptto “book”. The “dictionary” concept has all of the attributes of the“book” concept, and it is an instance of the concept which means that itcontains extra attributes. Students may understand the concept ingeneral but may be confused about a particular instance.

Example:

In the journalization activity, the concept transaction can be brokendown into two sections: the debit and the credit. And each of those canbe specialized into specialization categories, such as a credit to“Accounts payable”. Students may not be confused about debits but theinstance “Accounts Payable”.

Interface Constraints

Interface Constraint: Business Deliverable

When creating target group hierarchies, designers need to consider thetype of deliverable the student is creating. For each of the sections ofthe deliverable, the designer needs to create a target group. The targetgroups should contain an orderly structure, such as moving from top tobottom. Reviewing the deliverable in the order it is created structuresthe critique so that students know where to look next, after receivingfeedback. In the current Intelligent Tutoring Agent, this structuring offeedback around the student-created deliverable can be accomplished intwo ways. First, the designer can make every section of the deliverablea target. In addition, the designer can make some sections targets andsome modifying attributes. Modifying attributes can be remediated onspecifically, or in conjunction with the target.

In the journalization activity, the sections of the product—the journalentry—mirrors the concepts involved—debits and credits. But there are afew extra items on the journal which are (in most cases) not involved inthe main concepts being taught, and these are the dollar amounts to bejournalized. The dollar amounts which are journalized are associatedwith the journal entry as an attribute. Attributes modify the sourceitem (account name), which makes it possible to tell if the source itemis correct alone or with the attribute attached. As a designer, feedbackshould be created which takes all of this into account. Students shouldbe told if they have the journal entry correct and the amount wrong, orif they have the whole thing wrong.

Interface Constraint: Screen Space

Many times one concept will span many sections of the interface. It isimportant to group the target groups so that they are interfacespecific. Therefore, even though one product may span multipleinterfaces, the target groups should be centered around the interfacesso that the students receive feedback only about what they can see.

In the journalization activity, the sections of the deliverable—thecollection of journal entries in the ledger—span many separateinterfaces. Each source document must be seen individually. Therefore,some target groups are organized across all source documents—such as alldebits—and others are specific to the individual source documents—suchas that source document's debits. The target group's hierarchy mustinclude a section for across source documents—across interfaces—andthose within one source document—one interface.

Information Overload

As with any real-life tutor, you do not want to give too muchinformation for the student to digest at once. If there are twenty-fiveproblems, the tutor should not give feedback about all errorssimultaneously. Instead, the tutor should give feedback about just twoor three things which the student can correct before asking for morefeedback.

In the journalization activity, there are a limited number of targets onthe interface at one time—one debit and one credit. But if it were thewhole General Ledger, it could have too many pieces of feedback for thestudent to digest at once and could overwhelm the student. In this case,the designer should scale the feedback so that just a handful come backat once. This is best done by having small target groups defined, butcan also be done by identifying to the tutor how many different piecesof remediation are appropriate to deliver at one time.

Positive Reinforcement

In addition to creating target groups which are small in size, designersmay want to create target groups which evaluate the first few steps astudent makes. These early target groups will allow the student to seeif he is on track and understand the goal of the interaction. This is,in general, a good remediation strategy, but may not be relevant in alllearning situations.

In the journalization activity, there are twenty source documents tojournalize. Students should NOT be encouraged to ask for feedback atevery step, but when they have completed all of their work. This willensure that students try and learn all of the information first and notrely completely on the hints of the tutor. But, target groups definedfor just the first three entries allow for feedback and hints to beprovided at the onset of the task, diminishing once these entries arecorrect.

Sequencing the Target Group Hierarchy

For feedback to be as effective as possible, it needs to provide theright information at the right time. If feedback is given too early, itis confusing; if feedback is given too late it is frustrating. In theICAT, feedback is returned according to Target Groups. The tutor willlook at the highest target group, if there is no feedback in that targetgroup, the tutor will look at the children target groups in order ofpriority.

FIG. 39 is a block diagram illustrating the feedback hierarchy inaccordance with a preferred embodiment. In FIG. 39, the tutor will firstlook for any relevant feedback to be delivered in target group #1A. Ifthere is nothing there, then the tutor will look in the highestprioritized child target group-in the B tier. If there is nothing inthat target group, then the tutor will look in the highest child targetgroup of target group #1B which is target group #1C. Because the targetgroup priority determines where the tutor looks for feedback withintier, a great deal of thought needs to be given to what comprises atarget group and how they are structured. There are four guidingprinciples which will help structure target groups to provide the rightinformation at the right time and help the student make the most of theinformation provided.

Positive Reinforcement First

Designers should identify the first few components a student will tryand complete first and sequences them first. This target group willevaluate just the first few moves a student makes and will tell him howhe is doing and how to apply the knowledge gained from the first fewsteps to the rest of the work he has to do.

In the journalization activity, students need to have reinforcement thatthey are on the right track before trying all of the journal entries.Therefore, the first three are grouped together and students canfeedback on how they completed this sub-group before having to completethe rest.

Completing this subsection gives students the positive reinforcementthey need to complete the rest.

Easy Before Hard

If all of the target groups are of equivalent size, designers need tosequence easier concepts before more complicated concepts. By placingeasier concepts first, a student will gain confidence in theirunderstanding of the domain and in their ability to complete thedeliverable. In addition, most complicated concepts are built on easierones so that presenting easier concepts first will allow the student togain the experience they need to complete the most complicated concepts.In the journalization activity, two legged journal entries areinherently easier than three legged and four legged journal entries.Therefore when a designer must sequence target groups of equal size, thedesigner should sequence the two legged journal entries before the threeand four legged entries.

First Things First

Besides sequencing easier concepts before hard concepts, anotherstrategy is to sequence target groups in order that they need to becompleted. If completing one section of the deliverable is aprerequisite for completing another section of the deliverable, it makessense to sequence those targets first. In the journalization activity, asource document needs to be journalized in terms of the account name andin terms of the dollar amount. However, the account name must beidentified before the amount is entered. It makes no difference whetherthe dollar figure of the account is right or wrong, until the studenthas the correct account name.

Writing Feedback

Creating and structuring target group hierarchies determines what isevaluated and the order the feedback is returned. Once the hierarchy hasbeen created and structured, designers need to write feedback which willhelp the student complete his goal. Going back to the goals of the tutoras educator, feedback needs to accomplish the following goals:

Identify concepts students do not understand

Identify student mistakes

Prompt students to reflect on their mistakes

Reinforce correct concepts and ideas

These goals can be thought of in two sections. The first two areevaluative and the second two are instructive. Evaluation andinstruction are two of the three main components of a piece of feedbacktext. The third component is Scope. These three components are describedin more detailed below, beginning with Scope, as it is generally thefirst portion of a piece of feedback text.

What the Feedback is Evaluating (Scope)

The most important information feedback provides a student is what thetutor is reviewing. In most instances, the student will have completedlots of different actions before asking the tutor to review his work.Because the student has completed a lot of different actions, the tutorfirst needs to describe what portion of the activity or deliverable isbeing reviewed. There are generally three ways to scope what the tutoris reviewing.

All Work

The tutor is looking at everything the student did. Some instances whenfeedback should look at everything the student has done are praise levelfeedback and redirect level feedback. I looked at all of the journalentries and there are problems in many of them. Why don't you . . . .

A Localized Area of Work

The tutor is looking at a subset of work the student completed. Thegreatest use of localized scoping if focus feedback. The feedback isfocusing the student on one area of difficulty and asking him to correctit. I am looking at the first five journal entries you made, and hereare the first three problems I found. The first . . .

A Specific Problem or Error

The tutor is focusing on one error and/or problem and helping thestudent understand that error.

Specific problem scoping is good for classic mistakes a student may makeand the designer may want to remediate. In the first journal entry, youincorrectly debited Accounts Payable. Review that transaction . . .

How the Student Did (Evaluation)

The second section of the feedback text should describe how the studentdid. This is where the severity principle is applied and the feedback iseither redirect, focus, polish or praise.

Redirect

Redirect feedback is appropriate for very severe errors: severe mistakesand misconceptions. This degree of severity can be assessed aggregatelyby recognizing there are problems throughout the student's work or itcan be done specifically by recognizing some basic items are incorrect.

Example:

I am looking at the first five journal entries you made, and there areproblems in most of them. Why don't you . . . I am looking at the firstfive journal entries you made, and you have made some basic mistakeswith debits and credits. Why don't you . . .

Focus

Focus feedback is appropriate for localized mistakes or misconceptions.Focus level mistakes can be identified aggregately by identifying anarea in which there are a number of mistakes or specifically byidentifying that some of the building block ideas are wrong.

Example:

I am looking at the first five journal entries you made, and there areproblems in many of the debits. Why don't you . . .

I am looking at the first five journal entries you made, I see problemswhen transactions are “on account”. Why don't you . . .

Polish

Polish level feedback is for syntactic problems. Student understand themain ideas and have no local problems. There may be just one or twomistakes the student has made. Polish feedback should specify where themistake is.

Example:

I am looking at the first five journal entries you made, and the thirdjournal entry has the debit incorrect. Why don't you . . .

Praise

Praise level feedback is reserved for instances of “correctness”; thedeliverable is correct and ready to be used in the business.

Example:

I am looking at the first five journal entries you made, and they areall correct remember . . .

Mastermind

Mastermind feedback is reserved for instances where the student is nottrying to learn a topic but trying to cheat his way through byrepeatedly asking for feedback. The feedback needs to be written so thatthe student recognizes that the tutor wants more work completed beforeproviding feedback.

Example:

You have not changed much of your work since the last time you asked meto review it. Review . . .

Incomplete

Incomplete feedback is reserved for instances where the student has notcompleted all of the required work. It should be remembered thatsometimes it is desired to give substantive feedback before everythingis complete so the student learns the process and concepts before tryingto complete the whole deliverable.

Example:

You have not done all of your work. I would like you to try completingall journal entries before asking for my review.

What the Student Should Do Next (Instruction)

The final piece of information the student needs is what to do next. Thestudent knows what the tutor reviewed and knows how he performed. Theonly thing the student does not know is what to do next. The type ofinstruction does not have to correspond with the severity of the error.

The instructions can be mixed and matched with the type of error. Someof the actions a student could be asked to perform are as follows.

Review the General Concept

If the tutor recognizes that there are many errors throughout thedeliverable, the tutor may suggest that the student go through a reviewof the supporting materials provided to gain an understanding of theideas and skills needed to complete the task.

Example:

There are problems in many journal entries, why don't you review how tojournalize transactions and then review your journal entries.

Review a Section of the Student's Work

If the student has many errors in one section, then the tutor maysuggest that the student go and review that section of their work.

Example:

There are problems in the first five journal entries, why don't youreview them.

Review Work With a Hint

If there is a certain idea or concept which the tutor believes thestudent does not understand, then the tutor may give a hint in the formof a question or statement for the student to think about before tryingto fix the problems.

Example:

There are problems in the first five journal entries. It looks like youhave made some errors with the expense debits. Remember that expensesare not capitalized. Why don't you review the first five journal entrieslooking for journal entries which contained incorrect debits to expenseaccounts.

Review Work Looking for Type of Error

If there is a specific type of error that the student has madethroughout his work, then the tutor may tell the student the specifictype of error and ask him to go through his work correcting this error.

Example:

There are problems in the first five journal entries. You have switchedall of your journal entries on account debits. Why don't you go and fixthem.

Review Work Looking for Specific Error

If there is a specific error that the student has committed, the tutormay tell the student the if specific error committed and where the erroris.

Example:

There is a problem with your third journal entry. The debit should notbe “Accounts Payable.”

Review work because it is correct and the student will want to use thisanalysis technique in the future.

Example:

Your first three journal entries are correct. Remember that the majordistinction between paying for something “On Account” or in cash. Thisis a distinction you will need to make in the future.

Do More Work

If it can be determined that the student is simply asking for feedbackto “Cheat” his way through the course, feedback should be provided totell the student that he needs to try and correct many more entriesbefore receiving substantive feedback.

Example:

You have not changed much of your work since the last time you asked meto review it. Please review all of your journal entries and correct manyof them.

Complete Your Work

When it can be determined that all of the work which should be completeis not, the feedback needs to tell the student to complete the workrequired.

Example:

You have not completed all of your work. I would like you to trycompleting all journal entries before asking for my review.

Writing Levels of Feedback

Even with effective feedback, students will often make the same types ofmistakes again or in different situations. The question is what to tellthe student the second time he makes the same or similar mistakes. Weassume that telling the student the same thing over and over is not theright answer. Therefore instead of telling the student the same thing,the feedback cycles to a lower, or secondary, level. At this time, webelieve that three levels of feedback is appropriate for most instances.If the target group is particularly complex, however, additional levelsof feedback may be required.

First Level of Feedback

The first level of feedback should focus more on telling the studentwhat is wrong and letting the student try and figure it out on his own.Therefore using the paradigm described above, the student should be toldwhat the tutor is reviewing, how he did and asked to retry it orreferred to some reference which could be used to find the answer.

Example:

There are problems in many journal entries. Why don't you review how tojournalize transactions and then review your work.

Second Level of Feedback

The second level of feedback should give hints and provide pieces of thepuzzle. I can be assumed that students cannot figure out the problem ontheir own and need some help. It is appropriate at this point to ask thestudent to review their work with a specific hint in mind or with aquestion to think about. Also, if there are specific points in thereference system to review, this is the time to provide them.

Example:

There are problems in the first five journal entries. It looks like youhave made some errors with expense debits. Remember that expenses arenot capitalized. Why don't you review the first five journal entrieslooking for journal entries which contain incorrect debits to expenseaccounts.

Third Level of Feedback

The third level of feedback is appropriate for examples. Use theparameter substitution language to insert an example of an error theymade into the feedback. Walk the student through the thought process heshould use to solve the problem and provide and example of how they didthe work right and how they did the work wrong.

Example:

There are problems in many of your journal entries. It looks like youhave made some errors distinguishing between “on account” and “cash”credits. In particular, you characterized journal entry #12 as a cashpurchase when in fact it is an “on account” purchase. Remember billswhich are not paid immediately are paid on account.

Writing Rules

With the hierarchies created and sequenced and the feedback written, thedesigner is ready to write rules. Rules fire the particular pieces offeedback the student reads. To write effective rules, designers mustrealize the piece of feedback and the rule are one and the same. Theonly difference is the language used. The feedback is written in Englishand the rules are written as patterns.

Example Rule:

If the student has attempted all of the first three journal entries

And they all contain at least one mistake

Then provide feedback “In the first three journal entries you have madeat least one mistake in each. Why don't you review them and see if youcan find the mistakes.”

In the above example, the rules has two conditions (attempt all threejournal entries and have at least one mistake in each). The feedback isan explicit statement of that rule. The feedback states “In the firstthree journal entries you have made at least one mistake in each. Whydon't you review them and see if you can find any mistakes.”

The rule and the feedback are exactly the same. Keeping the rules andthe feedback tightly linked ensures that the student receives thehighest quality feedback. The feedback exactly explains the problem therules found. If the feedback is more vague than the rule, then thestudents will not understand the exact nature of the problem. Thefeedback will simply hint at it. If the feedback is more specific thanthe rule, students will become confused. The student may not have madethe specific error the feedback is referring to under the umbrella rule.

Types of Rules

Because the rules need to map to the feedback, there will be six typesof rules associated with the six types of feedback: Praise, Polish,Focus, and Redirect, along with Mastermind and Incomplete.

Praise

Praise rules need to look for one hundred percent correct and NO errors.If the rule does not explicitly look for no errors, the rule will alsofire when the student has all of the right answers but also some of thewrong ones.

If 100% of the targets in the first three journal entries are correct

And they all contain no mistakes

Then provide praise feedback

Praise rules can be applied in many places other than the highest tasklevel. Praise rules can fire for instances where a student got an itemright. In general, these rules should be written for any instance whichposes a difficult problem and a student may need reinforcement as to howto complete the process and complete the deliverable.

Polish

Polish rules need to fire when almost everything in the target group iscorrect and the student is making small and insignificant mistakes.

If 80%-99% of the targets in the first three journal entries are correct

And the first three journal entries have been tried

Then provide polish feedback

This polish rule shows two things. First, the rule is scoped so that itwill not fire when any of the first three journal entries have not beenattempted. In addition, the rule will not fire if all of the journalentries are 100% correct. With these boundaries in place the rule willonly fire when the student has attempted all of the first three journalentries and they are 80%-99% correct. Note: The determination of theexact percentages which must be correct to receive “polish” versus“focus” or “redirect” feedback will be determined by the designer, andare most likely specific to the particular task being completed.

Focus

Focus rules are the most common type of rule. Focus rules should firewhen the student knows enough to complete the task but not enough to getthe task correct.

If 40%-79% of the targets in the first three journal entries are correct

And the first three journal entries have been tried

Then provide focus feedback

This focus rule also shows scoping. The rules are scoped to fire between40% and 79%. Below 40% is redirect and above 79% is polish. The rulealso fires only when all of the required work has been attempted.

Redirect

Redirect rules should fire when it is clear that the student does nothave a good understanding of how to complete the task. This is evidencedby a significant number of errors found in the student's work.

If less than 40% of the first three journal entries are correct

And the first three journal entries have been tried.

Then provide redirect feedback

This redirect rule is to catch those who are truly lost. If the studenthas tried to complete all of the work, and they are less than 40%correct, then they need a great deal of help to continue the task.

Mastermind

Mastermind rules need to track down situations when the student issimply trying to cheat his way through the application.

If less than 40% of the first three journal entries are correct

And the student has made only one change twice in a row.

Then provide mastermind feedback

This mastermind rule catches those who are making one change, asking forfeedback over and over. One thing to keep in mind is that as a studentgets towards praise they need to make small changes and then ask forfeedback. To allow this, the above rule is scoped so that if the studenthas more than 40% of the work right the rule will not fire.

Incomplete

In many activities the student should try and complete most if not allof the work before asking for feedback. One of the goals of manytraining applications is to mimic the real world, and it is rare for anemployee to ask for a review after every little step they complete. Mostemployers want to see a significant amount of work done before askingfor a review.

If all of journal entries have NOT been tried,

Then provide incomplete feedback

Forcing a student to attempt all of his work first will require him togain confidence in his ability to complete the work. Therefore,incomplete rules should be used after baby-step feedback so thatstudents feel that they have the tools and ability to complete the wholetask before asking for feedback.

Principles of Rule Design

There are a couple of general rules which make rule creation andmaintenance easier.

Use Percentages Whenever Possible

It may seem easier at the time to write rules which look for specificnumbers of right and wrong items. But when a rule is written which looksfor a specific number, it means that if the data ever changes, you willneed to get back into that rule and tweak it so that it still fires atthe right time. It is far better to write percentage rules which firewhenever a certain percentage of work is either right or wrong. Then ifthe data ever changes and more right answers are added or some removed,then the rules may not need to be rewritten.

Scope the Rules as Tightly as Possible

As stated previously, it is very important to make the rules mirror thewritten feedback. If the feedback is vaguer than the rule, then thestudents will not understand the exact nature of the problem. Thefeedback will simply hint at it. If the feedback is more specific thanthe rule, students will become confused. The student may not have madethe specific error the feedback is referring to under the umbrella rule.

Data Dictionary in Accordance With a Preferred Embodiment

Column Type Len Description Domain Knowledge Model Data DictionarySource SourceID Counter Unique key for this table Source String  50 Nameof this object SourceDesc String 255 Documentation String that appearswith this object in auto-documentation reports SourceCaption String  50String that can be dynamically embedded into feedback text usingParameter Substitution Language (PSL) SourceItem SourceItemID CounterUnique key for this table SourceItem String  50 Name of this ObjectSourceItemDesc String 255 Documentation String that appears with thisobject in auto-documentation reports SourceItemText String  50 Stringthat Can be dynamically embedded into feedback text using ParameterSubstitution Language (PSL) TargetPage TargetPageID Counter Unique keyfor this table TargetPage String  50 Name of this object TargetPageDescString 255 Documentation String that appears with this object inauto-documentation reports TargetPageCaption String  50 String that Canbe dynamically embedded into feedback text using Parameter SubstitutionLanguage (PSL) Target TargetID Counter Unique key for this table TargetString  50 Name of this object TargetDesc String 255 DocumentationString that appears with this object in auto-documentation reportsTargetCaption String  50 String that Can be dynamically embedded intofeedback text using Parameter Substitution Language (PSL)SourceItemTarget SourceItemID Long SourceItemID of the associationTargetID Long TargetID of the association Relevance Float Value between−1 and 1 that indicates the relative relevance of this associationbetween a SourceItem and a Target. A negative value indicates that thisassociation is incorrect. A positive value indicates that it is correct.A value of zero indicates that this association is irrelevant. AttributeSourceItemID Long SourceItemID of the association TargetID Long TargetIDof the association AttributeID Counter Unique key for this tableAttribute String  50 Name of this object CorrectInd Bool Boolean valuethat indicates whether this Attribute is correct or incorrect for thisassociation of SourceItem and Target AttributeMin Double The lower boundfor the range of this attribute. AttributeMax Double The upper bound forthe range of this attribute. ControlSourceItem ModuleName String  50Name of module the control is on ControlName String  50 Name of Controlthe SourceItem is mapped to ItemNo Integer A single control may bemapped to multiple SourceItems depending on how it is viewed. If onecontrol is used on four different tabs to show four different values,the ItemNo will change as the tabs change, but the ControlName will staythe same. SourceItemID Long ID of SourceItem that this control is mappedto Start Integer For controls that contain text, this is the startposition of the text that the SourceItem is associated with. End IntegerFor controls that contain text, this is the end position of the textthat the SourceItem is associated with. TaskID Long This is the TaskIDthe module is in Description Text 255 Comment Information that canappear in the generated documentation reports. ControlTarget ModuleNameString  50 Name of module the control is ControlName String  50 Name ofControl the SourceItem is mapped to ItemNo Integer A single control maybe mapped to multiple Targets depending on how it is viewed. If onecontrol is used on four different tabs to show four different values,the ItemNo will change as the tabs change, but the ControlName will staythe same. TargetID Long ID of Target that this control is mapped toStart Integer For controls that contain text, this is the start positionof the text that the Target is associated with. End Integer For controlsthat contain text, this is the end position of the text that the Targetis associated with. TaskID Long This is the TaskID the module is inDescription Text 255 Comment Information that can appear in thegenerated documentation reports. Student Data Model Data DictionaryStudent SourceID Counter Unique key for this table Source String  50Name of this object SourceDesc String 255 Documentation String thatappears with this object in auto-documentation reports SourceCaptionString  50 String that Can be dynamically embedded into feedback textusing Parameter Substitution Language (PSL) StudentSubmissionSourceItemID Counter Unique key for this table SourceItem String  50Name of this Object SourceItemDesc String 255 Documentation String thatappears with this object in auto-documentation reports SourceItemTextString  50 String that Can be dynamically embedded into feedback textusing Parameter Substitution Language (PSL) UserSourceItemTargetSourceItemID Counter Unique key for this table SourceItem String  50Name of this Object SourceItemDesc String 255 Documentation String thatappears with this object in auto-documentation reports SourceItemTextString  50 String that Can be dynamically embedded into feedback textusing Parameter Substitution Language (PSL) Rule Model Data DictionaryRule TaskID Long ID of Task for which this rule is in scope CoachID LongID of Coach for which this rule is in scope RuleID Counter Unique keyfor this table Rule String  50 Name of this object RuleDesc String 255Documentation String that appears with this object in auto-documentationreports RuleCondCountMin Integer Minimum number of conditions that mustbe true for this Rule to fire RuleCondCountMax Integer Maximum number ofconditions that must be true for this Rule to fire CoachTopicID Long IDof CoachTopic that is activated when this rule fires RuleAggregateAndsRuleID Long ID of Rule of which this object is a condition RuleCondIDCounter Unique key for this table TargetGroupID Long ID of TargetGroupwhose aggregate values are compared to the aggregate boundaries of thiscondition AggRelevanceMin Float The TargetGroup's AggRelevanceMaxCalculated Aggregate Relevance must fall between this Min and Max forthis condition to be true AggUserCntPosMin Integer Thepositive-relevance AggUserCntPosMax associations the user has made usingTargets in this TargetGroup are counted to produce an Aggregate valuecalled ‘UserCntPos’. This TargetGroup's UserCntPos must fall betweenthis condition's AggUserCntPosMin and AggUserCntPosMax for thiscondition to be true. AggUserCntNegMin Integer The negative-relevanceAggUserCntNegMax associations the user has made using Targets in thisTargetGroup are counted to produce an Aggregate value called‘UserCntNeg’. This TargetGroup's UserCntNeg must fall between thiscondition's AggUserCntNegMin and AggUserCntNegMax for this condition tobe true. AggUserCntZeroMin Integer The zero-relevance AggUserCntZeroMaxassociations the user has made using Targets in this TargetGroup arecounted to produce an Aggregate value called ‘UserCntZero’. ThisTargetGroup's UserCntZero must fall between this condition'sAggUserCntZeroMin and AggUserCntZeroMax for this condition to be true.AggUserSumPosMin Float The relevance values of AggUserSumPosMax thepositive-relevance associations the user has made using Targets in thisTargetGroup are summed to produce an Aggregate value called‘UserSumPos’. This TargetGroup's UserSumPos must fall between thiscondition's AggUserSumPosMin and AggUserSumPosMax for this condition tobe true. AggUserSumNegMin Float The relevance values of AggUserSumNegMaxthe negative-relevance associations the user has made using Targets inthis TargetGroup are summed to produce an Aggregate value called‘UserSumNeg’. This TargetGroup's UserSumNeg must fall between thiscondition's AggUserSumNegMin and AggUserSumNegMax for this condition tobe true. AggUserCntPos2Min Integer The positive-relevanceAggUserCntPos2Max associations the user has made using Targets in thisTargetGroup where the user's Attribute are counted to produce anAggregate value called ‘UserCntPos2’. This TargetGroup's UserCntPos2must fall between this condition's AggUserCntPos2Min andAggUserCntPos2Max for this condition to be true. RuleSpecificMappingAnds RuleID Long ID of Rule of which this object is a conditionSourceItemID Long SourceItemID of the association TargetID Long TargetIDof the association SourceItemID Long Unique key for this tableAttributeMatchType Byte AttributeID Long Documentation String thatappears with this object in auto-documentation reportsAttributeMatchType AttributeMatchType Byte Unique key for this tableAttributeMatchTypeD String 255 Brief text description of esc eachAttributeMatchType Type Feedback Model Data Dictionary CoachTopic TaskIDLong ID of Task for which this object is in scope TargetGroupID Long IDof TargetGroup which this topic of remediation relates to CoachTopicIDCounter Unique key for this table CoachTopic String  50 Name of thisobject CoachTopicDesc String 255 Documentation String that appears withthis object in auto-documentation reports CoachTopicPriority String  3Priority of this CoachTopic with respect to other CoachTopics in thesame TargetGroup RemediationType String  50 Type of remediation thatthis CoachTopic is. This determines how the CoachTopic is handled atruntime. CoachItemStandA String  50 When all the Stand AloneloneReentrySeqID CoachItems in this CoachTopic have been used, they arerestarted on the CoachItemStandAloneReentry SeqID. If theCoachItemStandAloneReentry SeqID = 0 the StandAlone half of theCoachTopic is expired and no longer used. CoachItemChildR String  50When all the Child eentrySeqID CoachItems in this CoachTopic have beenused, they are restarted on the CoachItemChildReentrySeqID. If theCoachItemChildReentrySeqID = 0 the Child half of the CoachTopic isexpired and no longer used. RemediationType SourceItemID Counter Uniquekey for this table SourceItem String  50 Name of this ObjectSourceItemDesc String 255 Documentation String that appears with thisobject in auto-documentation reports SourceItemText String  50 Stringthat Can be dynamically embedded into feedback text using ParameterSubstitution Language (PSL) CoachItem SourceItemID Counter Unique keyfor this table SourceItem String  50 Name of this Object SourceItemDescString 255 Documentation String that appears with this object inauto-documentation reports SourceItemText String  50 String that Can bedynamically embedded into feedback text using Parameter SubstitutionLanguage (PSL)

Source Code In Accordance With A Preferred Embodiment////////////////////////////////////////////////////////////////////////////////////////// tutxport.h////////////////////////////////////////////////////////////////////////////////////////// Control Functions /* ***************************************** *Name: TuResumeStudent * Purpose: To Resume a Student In progress. *Author: Mike Smialek/Andersen Consulting * Input * Parameters: longStudentID * The Unique ID of the Student to load * * long TaskID * TheUnique ID of the Task to Load * * int fromSubmissionSeqID * TheSubmission from which the Student continues the Task * <0 :Resume Taskfrom latest submission * =0 :Restart Task * >0 :Continue from a specificsubmission * * Output * Parameters: none * Function Return * Variables:TUT_ERR_DB_COULDNT_OPEN_DATABASE * TUT_ERR_DOC_COULDNT_LOAD_TASK_DOC *TUT_ERR_LOD_NO_COACHTOPICS_FOUND * TUT_ERR_LOD_NO_COACHITEMS_FOUND *TUT_ERR_LOD_NO_COACHES_FOUND * TUT_ERR_LOD_NO_SOURCEITEMTARGETS_FOUND *TUT_ERR_LOD_NO_SOURCES_FOUND * TUT_ERR_LOD_NO_SOURCEITEMS_FOUND *TUT_ERR_LOD_NO_TARGETGROUPS_FOUND * TUT_ERR_LOD_NO_TARGETS_FOUND *TUT_ERR_LOD_NO_TARGETPAGES_FOUND *TUT_ERR_LOD_NO_TARGETGROUPTARGETS_FOUND * TUT_ERR_LOD_NO_RULES_FOUND *TUT_ERR_DB_COULDNT_OPEN_RECORDSET * * TUT_ERR_OK * * Notes: Loads fromDatabase or Document based on values * of m_StorageTypeTask andm_StorageTypeStudent * ***************************************** */extern “C” { long _export WINAPI TuResumeStudent(long StudentID, longTaskID, int fromSubmissionSeqID ); // Resumes a Student's work for theTask at the specified Submission } extern “C” { long _export WINAPITuLoadArchivedSubmissions(long StudentID, long TaskID, intfromSubmissionSeqID, int toSubmissionSeqID); // Loads ArchivedSubmissions For a Student's work in a Task } extern “C” { long _exportWINAPI TuUseArchivedSubmissions(int n); // Replays n Archivedsubmissions for debugging } extern “C” { long _export WINAPITuSaveCurrentStudent(); // Saves Current Student's work to DB } extern“C” { long _export WINAPI TuSimulateStudent(long StudentID, long TaskID,float Intelligence, float Tenacity, int MaxTurns); // Not operational }extern “C” { long _export WINAPI TuWriteUserDebugInfo(); // writesactive CoachTopics to DB for Debugging } extern “C” { long _exportWINAPI KillEngine( long 1TaskID); // Delete all Dynamic objects beforeshutdown } /* ***************************************** * Name:LoadTaskInfo * Purpose: To load data for a Task only. Student data isnot loaded * Author: Mike Smialek/Andersen Consulting * Input *Parameters: long TaskID * The Unique ID of the Task to Load * Output *Parameters: none * * Function Return * Variables:TUT_ERR_DB_COULDNT_OPEN_DATABASE * TUT_ERR_DOC_COULDNT_LOAD_TASK_DOC *TUT_ERR_LOD_NO_COACHTOPICS_FOUND * TUT_ERR_LOD_NO_COACHITEMS_FOUND *TUT_ERR_LOD_NO_COACHES_FOUND * TUT_ERR_LOD_NO_SOURCEITEMTARGETS_FOUND *TUT_ERR_LOD_NO_SOURCES_FOUND * TUT_ERR_LOD_NO_SOURCEITEMS_FOUND *TUT_ERR_LOD_NO_TARGETGROUPS_FOUND * TUT_ERR_LOD_NO_TARGETS_FOUND *TUT_ERR_LOD_NO_TARGETPAGES_FOUND *TUT_ERR_LOD_NO_TARGETGROUPTARGETS_FOUND * TUT_ERR_LOD_NO_RULES_FOUND *TUT_ERR_DB_COULDNT_OPEN_RECORDSET * * TUT_ERR_OK * Notes:***************************************** */ extern “C” { long _exportWINAPI LoadTaskInfo( longITaskID); // Clear and (re)load info for TaskID} /* ***************************************** * * Name: TuLoadTaskDoc *Purpose: Loads a Tutor Document containing Task Data * Author: MikeSmialek/Andersen Consulting * Input * Parameters: long 1TaskID *TaskIDToLoad * Output * Parameters: none * * Function Return *Variables: TUT_ERR_DOC_COULDNT_LOAD_TASK_DOC *TUT_ERR_LOD_NO_COACHTOPICS_FOUND * TUT_ERR_LOD_NO_COACHITEMS_FOUND *TUT_ERR_LOD_NO_COACHES_FOUND * TUT_ERR_LOD_NO_SOURCEITEMTARGETS_FOUND *TUT_ERR_LOD_NO_SOURCES_FOUND * TUT_ERR_LOD_NO_SOURCEITEMS_FOUND *TUT_ERR_LOD_NO_TARGETGROUPS_FOUND * TUT_ERR_LOD_NO_TARGETS_FOUND *TUT_ERR_LOD_NO_TARGETPAGES_FOUND *TUT_ERR_LOD_NO_TARGETGROUPTARGETS_FOUND * TUT_ERR_LOD_NO_RULES_FOUND * *TUT_ERR_OK * * Notes: TaskID is used to format the file name of theDocument. * ***************************************** */ extern “C” {long _export WINAPI TuLoadTaskDoc( long 1TaskID ); // Clear and (re)loadinfo for TaskID from TaskDoc } */***************************************** * Name: TuSaveTaskDoc *Purpose: Saves The Task data as a Tutor Document * Input * Parameters:long lTaskID * TaskIDToSave * Output * Parameters: none * * FunctionReturn * Variables: TUT_ERR_DOC_COULDNT_SAVE_TASK_DOC * * TUT_ERR_OK * *Notes: TaskID is currently only used to format the file name of theDocument. * If a TaskID is passed in that is different than the loadedTask, * it will save the loaded data as if it were data for Task ID***************************************** */ extern “C” { long _exportWINAPI TuSaveTaskDoc( long lTaskID ); // Save info for TaskID intoTaskDoc } /* ***************************************** * Name: TuGo *Purpose: Kicks off Submission or Secret Submission * Input * Parameters:long lCoachID * CoachID submitting to * >0 :Submit to Specific Coach *=0 :Secret Submission to all Coaches * Output * Parameters: none * *Function Return * Variables: TUT_ERR_OK * * Notes:***************************************** */ extern “C” { long _exportWINAPI TuGo( long lCoachID); // kick off algorithm } /****************************************** * Name: TuIsDirty * Purpose:Gets the Dirty Status of the Task or of an individual Coach * Input *Parameters: long lCoachID * CoachID for which to determine DirtyStatus * >0 :Determines Dirty Status for specific Coach * =0 :DeterminesDirty Status for whole Task * Output * Parameters: LPINT IsDirty * TRUEindicates this Coach or Task is Dirty * FALSE indicates this Coach orTask is not Dirty * If one or more Coaches is dirty the Task is Dirty *Function Return * Variables: TUT_ERR_LOD_NO_COACHES_FOUND *TUT_ERR_LOD_COACHID_NOT_FOUND * TUT_ERR_OK * Notes:***************************************** */ extern “C” { long _exportWINAPI TuIsDirty(long TaskID, long lCoachID, LPINT IsDirty); } /****************************************** * Name: TuGetSubmissionSeqID *Purpose: Returns the current SubmissionSeqID * Author: Mike Smialek /Andersen Consulting * Input * Parameters: long TaskID * The TaskID forwhich you want the SubmissionSeqID * Output * Parameters: none * *Function Return * Variables: SubmisisonSeqID of the currentSubmission * * Notes: ***************************************** */extern “C” { long _export WINAPI TuGetSubmissionSeqID(long TaskID); } /****************************************** * Name:TuGetFeedbackPrevCoachID * Purpose: Returns the CoachID of The CoachThat delivered the previous feedback * Function Return * Variables:CoachID of The Coach That delivered the previous feedback * Notes:***************************************** */ extern “C” { long _exportWINAPI TuGetFeedbackPrevCoachID(); } /****************************************** * Name: TuGetApprovalStatus *Purpose: Gets the Approval Status of the Task or of an individualCoach * Input * Parameters: long lCoachID * CoachID for which todetermine Approval * >0 :Determines approval for specific Coach * =0:Determines approval for whole Task * Output * Parameters: LPINTApprovalRequired * TRUE indicates this Coach or Task requires approval *FALSE indicates this Coach or Task does not require approval * (AlwaysTRUE when input CoachID = 0) * * LPINT Approved * TRUE indicates thisCoach or Task is approved * FALSE indicates this Coach or Task is notapproved * * Function Return * Variables: TUT_ERR_LOD_NO_COACHES_FOUND *TUT_ERR_LOD_COACHID_NOT_FOUND * * TUT_ERR_OK * * Notes:***************************************** */ extern “C” { long _exportWINAPI TuGetApprovalStatus( long lCoachID, LPINT ApprovalRequired, LPINTApproved ); // return approval status for CoachID } /****************************************** * Name: TuCanProceed *Purpose: Determines ifTask is in state in which user can proceed toanother Task * Input * Parameters: long lTaskID * TaskID to examine *Output * Parameters: LPINT CanProceed * TRUE indicates user can proceedfrom this Task * FALSE indicates user can not proceed from this Task *Function Return * Variables: TUT_ERR_LOD_NO_COACHES_FOUND * *TUT_ERR_OK * Notes: ***************************************** */ extern“C” { long _export WINAPI TuCanProceed( long lTaskID, LPINT CanProceed);} /* ***************************************** * Name: TuMenu * Purpose:Opens Menu Dialog * Author: Mike Smialek / Andersen Consulting * Input *Parameters: none * Output * Parameters: none * Function Return *Variables: TUT_ERR_OK * Notes: ******************************************/ extern “C” { long _export WINAPI TuMenu(); } /****************************************** * Name: TuTesterComment *Purpose: Opens Tester Comment Dialog * Input * Parameters: none *Output * Parameters: none * Function Return * Variables: TUT_ERR_OK *Notes: ***************************************** */ extern “C” { long_export WINAPI TuTesterComment(); }////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// Notification Functions /* ***************************************** *Name: TuCreateMap * Purpose: To Create an association between aSourceItem and a Target * with a modifying Attribute value * Input *Parameters: long SIID * SourceItemID of existing association tocreate * * long TID * TargetID of association to create * * doubleAttr * Attribute value of association to create * * Output * Parameters:none * * Function Return * Variables:TUT_ERR_TUF_USIT_TARGET_NOT_FOUND * TUT_ERR_TUF_USIT_DUPLICATE_FOUND * *TUT_ERR_OK * Notes: ***************************************** */ extern“C” { long _export WINAPI TuCreateMap( long SIID, long TID, double Attr); } /* ***************************************** * Name: TuModifyMap *Purpose: To Modify an association between a SourceItem and a Target *with a new modifying Attribute value * Input * Parameters: long SIID *SourceItemID of existing association to Modify * long TID * TargetID ofexisting association to Modify * double Attr * New Attribute value forassociation * Output * Parameters: none * Function Return * Variables:TUT_ERR_TUF_USIT_TARGET_NOT_FOUND * TUT_ERR_TUF_USIT_DUPLICATE_FOUND * *TUT_ERR_OK * * Notes: This function calls TuDeleteMap / TuCreateMap ****************************************** */ extern “C” { long _exportWINAPI TuModifyMap( long SIID, long TID, double Attr ); } /****************************************** * Name: TuDeleteMap2 *Purpose: To Delete an association between a SourceItem and a Target *Input * Parameters: long SIID * SourceItemID of association todelete * * long TID * TargetID of association to delete * * doubleAttr * Attribute value of association to delete * Output * Parameters:none * Function Return * Variables: TUT_ERR_TUF_USIT_TARGET_NOT_FOUND *TUT_ERR_TUF_USIT_NOT_FOUND * * TUT_ERR_OK * * Notes: This functionignores the Attribute value and calls * TuDeleteMap( long SIID, longTID) ***************************************** */ extern “C” { long_export WINAPI TuDeleteMap2( long SIID, long TID, double Attr); } /****************************************** * Name: TuDeleteMap * Purpose:To Delete and association between a SourceItem and a Target * Input *Parameters: long SIID * SourceItemID of association to delete * * longTID * TargetID of association to delete * * Output * Parameters:none * * Function Return * Variables:TUT_ERR_TUF_USIT_TARGET_NOT_FOUND * TUT_ERR_TUF_USIT_NOT_FOUND *TUT_ERR_OK * Notes: ***************************************** */ extern“C” { long _export WINAPI TuDeleteMap( long SIID, long TID ); }////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// Configuration Functions /* * Name: TuSetODBCConnect * Purpose: To setODBC Connect String for the Task Data Database * Input * Parameters:LPCSTR ODBCConnect * ODBC Connect String for the Task Data Database *Output * Parameters: none * * Function Return * Variables:TUT_ERR_OK * * Notes: ***************************************** */extern “C” { long _export WINAPI TuSetODBCConnect( LPCSTR ODBCConnect );} /* ***************************************** * Name:TuSetODBCConnectTrack * Purpose: To set ODBC Connect String for theStudent Tracking Database * Input * Parameters: LPCSTR ODBCConnect *ODBC Connect String for the Student Tracking Database * Output *Parameters: none * Function Return * Variables: TUT_ERR_OK * Notes:***************************************** */ extern “C” { long _exportWINAPI TuSetODBCConnectTrack( LPCSTR ODBCConnect ); } /****************************************** * Name: TuSetTaskDocPathName *Purpose: To set path and name of the Task Document file * Input *Parameters: LPCSTR fnm * Path and name of the Task Document file *Output * Parameters: none * * Function Return * Variables:TUT_ERR_OK * * Notes: ***************************************** */extern “C” { long _export WINAPI TuSetTaskDocPathName( LPCSTR fnm ); }/* ***************************************** * Name:TuSetFeedbackFileName * Purpose: To set path and name of file to use forholding feedback * Input * Parameters: LPCSTR fnm * Path and name offile to use for holding feedback * Output * Parameters: none * *Function Return * Variables: TUT_ERR_OK * * Notes:***************************************** */ extern “C” { long _exportWINAPI TuSetFeedbackFileName( LPCSTR fnm ); } /****************************************** * Name:TuSetFeedbackPrevFileName * Purpose: To set path and name of file to usefor holding previous feedback * Input * Parameters: LPCSTR fnm * Pathand name of file to use for holding previous feedback * Output *Parameters: none * * Function Return * Variables: TUT_ERR_OK * Notes:***************************************** */ extern “C” { long _exportWINAPI TuSetFeedbackPrevFileName( LPCSTR fnm ); /****************************************** * Name: TuSetLogFileName *Purpose: To set path and name of file to use for full logging * Input *Parameters: LPCSTR fnm * Path and name of file to use for full logging *Output * Parameters: none * * Function Return * Variables: TUT_ERR_OK *Notes: ***************************************** */ extern “C” { long_export WINAPI TuSetLogFileName( LPCSTR fnm ); } /****************************************** * Name: TuSetLogLoadFileName *Purpose: To set path and name of file to use for load logging * Input *Parameters: LPCSTR fnm * Path and name of file to use for load logging *Output * Parameters: none * * Function Return * Variables:TUT_ERR_OK * * Notes: ***************************************** */extern “C” { long _export WINAPI TuSetLogLoadFileName( LPCSTR fnm ); }/* ***************************************** * Name:TuSetLogStudentFileName * Purpose: To set path and name of file to usefor student logging * Input * Parameters: LPCSTR fnm * Path and name offile to use for student logging * Output * Parameters: none * * FunctionReturn * Variables: TUT_ERR_OK * * Notes:***************************************** */ extern “C” { long _exportWINAPI TuSetLogStudentFileName( LPCSTR fnm ); } /****************************************** * Name:TuSetLogSubmissionFileName * Purpose: To set path and name of file touse for submission logging * Input * Parameters: LPCSTR fnm * Path andname of file to use for submission logging * Output * Parameters:none * * Function Return * Variables: TUT_ERR_OK * * Notes:***************************************** */ extern “C” { long _exportWINAPI TuSetLogSubmissionFileName( LPCSTR fnm ); } /****************************************** * Name: TuSetLogErrFileName *Purpose: To set path and name of file to use for error logging * Input *Parameters: LPCSTR fnm * Path and name of file to use for errorlogging * Output * Parameters: none * * Function Return * Variables:TUT_ERR_OK * Notes: ***************************************** */ extern“C” { long _export WINAPI TuSetLogErrFileName( LPCSTR fnm ); } /****************************************** * Name: TuSetTrace * Purpose:To turn Trace on and off * Input * Parameters: int TraceStatus *TUT_TRACE_ON :Turn Trace On * TUT_TRACE_OFF :Turn Trace Off * Output *Parameters: none * * Function Return * Variables: Previous Trace StatusValue * TUT_TRACE_ON * TUT_TRACE_OFF * * TUT_ERR_INVALID_TRACE_STATUS *Notes: ***************************************** */ extern “C” { long_export WINAPI TuSetTrace( int TraceStatus ); } /****************************************** * Name: TuSetTrack * Purpose:To turn Tracking on and off. While tracking is on * all work the userdoes and all feedback the user receives * is kept. While Tracking is offonly the most recent work is kept. * Input * Parameters: intTrackStatus * TUT_TRACK_ON :Turn Tracking On * TUT_TRACK_OFF :TurnTracking Off * Output * Parameters: none * Function Return * Variables:Previous Trace Status Value * TUT_TRACK_ON * TUT_TRACK_OFF * *TUT_ERR_INVALID_TRACK_STATUS * Notes:***************************************** */ extern “C” { long _exportWINAPI TuSetTrack( int TrackStatus ); } /****************************************** * Name:TuSetShowExceptionPopup * Purpose: To Exception popups on and off. *Input * Parameters: int PopupStatus * TUT_POPUP_ON :Turn Exceptionpopups On * TUT_POPUP_OFF :Turn Exception popups Off * Output *Parameters: none * * Function Return * Variables: Previous Exceptionpopup Status Value * TUT_POPUP_ON * TUT_POPUP_OFF * *TUT_ERR_INVALID_POPUP_STATUS * Notes:***************************************** */ extern “C” { long _exportWINAPI TuSetShowExceptionPopup( int PopupStatus ); } /****************************************** * Name: TuSetStorageType *Purpose: To Direct Task and Student data to be loaded and saved * usinga Document or Database * Input * Parameters: long StorageTypeTask *TUT_STORAGE_TYPE_DOCUMENT :Load and Save Task Data using Document *TUT_STORAGE_TYPE_DATABASE :Load and Save Task Data using Database * *long StorageTypeStudent * TUT_STORAGE_TYPE_DOCUMENT :Load and SaveStudent Data using Document * TUT_STORAGE_TYPE_DATABASE :Load and SaveStudent Data using Database * Output * Parameters: none * * FunctionReturn * Variables: TUT_ERR_INVALID_STORAGE_TYPE_TASK *TUT_ERR_INVALID_STORAGE_TYPE_STUDENT * * TUT_ERR_OK * Notes:***************************************** */ extern “C” { long _exportWINAPI TuSetStorageType( int StorageTypeTask, int StorageTypeStudent );}////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

Simulation Engine

The idea is for the designer to model the task that he wants a studentto accomplish using an Excel spreadsheet. Then, have an algorithm orengine that reads all the significant cells of the spreadsheet andnotifies the Intelligent Coaching Agent with the appropriate information(SourceItemID, TargetID and Attribute). This way, the spreadsheet actsas a central repository for student data, contains most of thecalculations required for the task and in conjunction with the enginehandles all the communication with the ICA. The task is self containedin the spreadsheet, therefore the designers no longer need a graphicaluser interface to functionally test their designs (smart spreadsheet).FIG. 40 is a block diagram illustrating how the simulation engine isarchitected into a preferred embodiment of the invention.

Once the model and feedback for it are completely tested by designers,developers can incorporate the spreadsheet in a graphical userinterface, e.g., Visual Basic as a development platform. The simulationspreadsheet is usually invisible and populated using functions providedby the engine. It is very important that all modifications that the ICAneeds to know about go through the engine because only the engine knowshow to call the ICA. This significantly reduced the skill level requiredfrom programmers, and greatly reduced the time required to program eachtask. In addition, the end-product was less prone to bugs, because thetutor management was centralized. If there was a tutor problem, we onlyhad to check on section of code. Finally, since the simulation engineloaded the data from a spreadsheet, the chance of data inconsistencybetween the tutor and the application was nil.

Object Model

FIG. 41 is a block diagram setting forth the architecture of asimulation model in accordance with a preferred embodiment. TheSimulation Object Model consists of a spreadsheet model, a spreadsheetcontrol object, a simulation engine object, a simulation database, inputobjects, output objects, list objects and path objects.

Spreadsheet Object

The first object in our discussion is the Spreadsheet object. TheSpreadsheet is the support for all simulation models. A control objectthat is readily integrated with the Visual Basic development plat. Thecontrol supports printing and is compatible with Microsoft Excelspreadsheets. With that in mind, designers can use the power of Excelformulas to build the simulation. The different cells contained in thespreadsheet model can be configured as Inputs, Outputs or Lists andbelong to a simulation Path.

Input Object

All cells in the spreadsheet that need to be manually entered by thedesigner or the student via the GBS application are represented by inputobjects. Every input has the following interface:

Field Name Data Type Description InputID long Primary Key for the tableTaskID long TaskID of the task associated with the input PathID longPathID of the path associated with the input InputName string*50 Name ofthe input InputDesc string*255 Description of the input ReferenceNamestring*50 Name of the spreadsheet cell associated with the inputTutorAware boolean Whether the ICA should be notified of any changes tothe input SourceItemID long SourceItemID if input is a distinct input 0if input is a drag drop input TargetID long TargetID of the input Rowlong Spreadsheet row number of the input → speed optimization Columnlong Spreadsheet column number of the input → speed optimizationSheetName string*50 Sheet name were the input is located → speedoptimization

This information is stored for every input in the Input table of thesimulation database (ICASim.mdb). Refer to the example below.

When designers construct their simulation model, they must be aware ofthe fact that there are 2 types of Inputs:

1. Distinct Input

2. Drag & Drop Input

Distinct Input

The Distinct Input consists of a single spreadsheet cell that can befilled by the designer at design time or by the GBS application at runtime via the simulation engine object's methods. The purpose of the cellis to provide an entry point to the simulation model. This entry pointcan be for example an answer to a question or a parameter to anequation. If the cell is TutorAware (all inputs are usually TutorAware),the ICA will be notified of any changes to the cell. When the ICA isnotified of a change two messages are in fact sent to the ICA:

1. An ICANotifyDestroy message with the input information i.e.,SourceItemID, TargetID and null as Attribute. This message is to advisethe ICA to remove this information from its memory.

2. An ICANotifyCreate message with the input information i.e.,SourceItemID, TargetID, Attribute (cell numeric value). This message isto advise the ICA to add this information to its memory.

A Distinct Input never requires that a user answer a mathematicsquestion. These are the steps required to configure that simulation.FIG. 42 illustrates the arithmetic steps in accordance with a preferredembodiment.

1. Define a name for cell C2 in Excel. Here we have defined “Distinct_Input”.

2. In the ICA, define a task that will be assigned to the simulation.Ex: a TaskID of 123 is generated by the ICA.

3. In the ICA, define a Target for the input. Ex: a TargetID of 4001 isgenerated by the ICA.

4. In the ICA, define a SourceItem for the input. Ex: a SourceItemID of1201 is generated by the ICA.

5. Associate the input to a path (refer to Path object discussion).

6. Add the information in the Input table of the simulation enginedatabase.

A record in an Input table is presented below.

InputID: 12345 TaskID: 123 PathID: 1234 InputName: Question 1 inputInputDesc: Distinct input for Question 1 ReferenceName: Distinct_InputTutorAware: True SourceItemID 1201 TargetID: 4001 Row: 2 Column: 3SheetName: Sheet1

The Row, Column and SheetName are filled in once the user clicks “RunInputs/Outputs”. The simulation engine decodes the defined name(Reference Name) that the designer entered, and populates the tableaccordingly. This is an important step. We had several occasions when adesigner would change the layout of a spreadsheet, i.e., move a definedname location, then forget to perform this step. As such, bizarre datawas being passed to the tutor; whatever data happened to reside in theold row and column. Once the configuration is completed, the designercan now utilize the ICA Utilities to test the simulation.

Drag & Drop Input

The drag & drop input consist of two consecutive spreadsheet cells. Bothof them have to be filled by the designer at design time or by the GBSapplication at run time via the simulation engine object's methods. Thistype of input is used usually when the user must choose one answer amonga selection of possible answers. Drag & drop inputs are alwaysTutorAware. The left most cell contains the SourceItemID of the answerpicked by the user (every possible answer needs a SourceItemID) and therightmost cell can contain a numeric value associated to that answer.You need to define a name or ReferenceName in the spreadsheet for therightmost cell.

ICA will be notified of any changes to either one of the cells. When theICA is notified of a change two messages are in fact sent to the ICA:

1. An ICANotifyDestroy message with the input information i.e.,SourceItemID before the change occurred, TargetID of the input and theAttribute value before the change occurred.

2. An ICANotifyCreate message with the input information i.e.,SourceItemID after the change occurred, TargetID of the input and theAttribute value after the change occurred.

Let's demonstrate the use of a drag & drop input building on top of theprevious example. Here, the user is asked to answer yet anothermathematics question. These are the steps required to configure thatsection of the simulation. FIG. 43 illustrates a drag & drop inputoperation in accordance with a preferred embodiment.

1. Define a name for cell C11 in Excel. Here we have defined “DragDrop_Input”.

2. Let's use the same TaskID as before since Question 2 is part of thesame simulation as Question 1. Ex: TaskID is 123.

3. In the ICA, define a Target for the input. Ex: a TargetID of 4002 isgenerated by the ICA.

4. In the ICA, define a SourceItem for every possible answer to thequestion. Ex: SourceItemIDs 1202 to 1205 are generated by the ICA.

5. Associate the input to a path (refer to Path object discussion).

6. Add the information in the Input table of the simulation enginedatabase.

A record in the Input table in accordance with a preferred embodiment ispresented below.

InputID: 12346 TaskID: 123 PathID: 1234 InputName: Question 2 inputInputDesc: Drag & Drop input for Question 2 ReferenceName:DragDrop_Input TutorAware: True SourceItemID 0*** TargetID: 4002 Row: 11Column: 3 SheetName: Sheet1

List object

FIG. 44 illustrates list object processing in accordance with apreferred embodiment. The list object consists of one cell identifyingthe list (cell #1) and a series of placeholder rows resembling drag &drop inputs (cells #1.1-1.n to cells #n.1-n.n). The list is used usuallywhen the user must choose multiple elements among a selection ofpossible answers. Cell #1 must have a uniquely defined name also calledthe list name. Cells #1.1 to #n. 1 can contain the SourceItemID of onepossible answer picked by the user (every possible answer needs aSourceItemID). The content of these cells must follow this format:˜ListName˜SourceItemID. Cells #1.2 to #n.2 will hold the numeric value(attribute) associated with the SourceItemID in the cell immediately tothe left. Cells #1.3-1.n to #n.3-n.n are optional placeholders for dataassociated with the answer. KEY NOTE: When implementing a list objectthe designer must leave all the cells under #n.1 to #n.n blank becausethis range will shift up every time an item is removed from the list.

Every list has the following interface: Field Name Data Type DescriptionListID long Primary Key for the table TaskID long TaskID of the taskassociated with the list PathID long PathID of the path associated withthe list ListName string*50 Name of the list ListDesc string*255Description of the list ReferenceName string*50 Name of the spreadsheetcell associated with the list TutorAware boolean Whether the ICA shouldbe notified of any changes to the list TargetID long TargetID of theoutput TotalColumns long Total number of data columns Row longSpreadsheet row number of the output → speed optimization Column longspreadsheet column number of the output → speed optimization SheetNamestring*50 Sheet name were the input is located → speed optimization

Use of a list is demonstrated by continuing our math test. The mathquestion in this example invites the user to select multiple elements toconstruct the answer. These are the steps required to configure thatsection of the simulation. FIG. 45 illustrates the steps for configuringa simulation in accordance with a preferred embodiment.

1. Define a name for cell C23 in Excel. Here we have defined “The_List”.

2. Let's use the same TaskID as before since Question 3 is part of thesame simulation as Question 1 and 2. Ex: TaskID is 123.

3. In the ICA, define a Target for the list. Ex: a TargetID of 4006 isgenerated by the ICA.

4. In the ICA, define a SourceItem for every item that could be placedin the list. Ex: the following SourceItemIDs 1209, 1210, 1211, 1212,1213, 1214 are generated by the ICA.

5. Associate the list to a path (refer to Path object discussion).

6. Add the information in the List table of the simulation enginedatabase.

A record in the List table in accordance with a preferred embodiment ispresented in the table appearing below.

ListID: 12346 TaskID: 123 PathID: 1234 ListName: Question 3 listListDesc: List for Question 3 ReferenceName: The_List TutorAware: TrueTargetID: 4006 TotalColumns: 1 Row: 23 Column: 3 SheetName: Sheet1

Output Object

All cells in the spreadsheet that are result of calculations (do notrequire any external input) can be represented by output objects. Everyoutput has an interface as outlined in the table below.

Field Name Data Type Description OutputID long Primary Key for the tableTaskID long TaskID of the task associated with the output PathID longPathID of the path associated with the output OutputName string*50 Nameof the output OutputDesc string*255 Description of the outputReferenceName string*50 Name of the spreadsheet cell associated with theoutput TutorAware boolean Whether the ICA should be notified of anychanges to the output SourceItemID long SourceItemID of the outputTargetID long TargetID of the output Row long Spreadsheet row number ofthe output → speed optimization Column long Spreadsheet column number ofthe output → speed optimization SheetName string*50 Sheet name were theinput is located → speed optimization

All this information is stored for every output in the Output table ofthe simulation database (ICASim.mdb). When designers construct theirsimulation model, they must be aware of the fact that there is only 1type of Outputs: the Distinct Output. Distinct Output

A Distinct Output consists of one and only one spreadsheet cell thatcontains a formula or a result of calculations. The existence of Outputcells is the main reason to have a simulation model. If the cell isTutorAware, the ICA will be notified of any changes to the cell when alloutputs are processed otherwise the ICA will be unaware of any changes.When the ICA is notified of a change two messages are in fact sent tothe ICA:

1. An ICANotifyDestroy message with the output information i.e.,SourceItemID, TargetID and null as Attribute. This message is to advisethe ICA to remove this information from its memory.

2. An ICANotifyCreate message with the output information i.e.,SourceItemID, TargetID, Attribute (cell numeric value). This message isto advise the ICA to add this information to its memory. As opposed toDistinct Inputs and Drag & Drop Inputs which notify the ICA on everychange, Distinct Outputs are processed in batch just before asking theICA for feedback.

To notify the ICA of the total dollar amount of the items in the list.We definitely need a Distinct Output for that. The output will contain asum formula. FIG. 46 illustrates a distinct output in accordance with apreferred embodiment. The steps required to configure that section ofthe simulation taking in consideration that the list is alreadyconfigured are presented below.

1. Define a name for cell C24 in Excel. Here we have defined “Distinct_Output”.

2. Let's use the same TaskID as before since Question 3 is part of thesame simulation as Question 1 and 2. Ex: TaskID is 123.

3. In the ICA, define a Target for the output. Ex: a TargetID of 4005 isgenerated by the ICA.

4. In the ICA, define a SourceItem for the output. Ex: a SourceItemID of1215 is generated by the ICA.

5. Associate the output to a path (refer to Path object discussion).

6. Add the information in the Output table of the simulation enginedatabase.

A record in an Output table in accordance with a preferred embodiment ispresented below.

OutputID: 12347 TaskID: 123 PathID: 1234 OutputName: Question 3 outputOutputDesc: Distinct Output for Question 3 ReferenceName:Distinct_Output TutorAware: True SourceItemID 1215 TargetID: 4005 Row:24 Column: 6 SheetName: Sheet1

Path Object

Paths are used to divide a simulation model into sub-Simulations meaningthat you can group certain inputs, outputs and lists together to form acoherent subset or path. Every path has the following interface:

Field Name Data Type Description PathID long Primary Key for the tableTaskID long TaskID of the task associated with the path PathNo longNumeric value associated to a path PathName string*50 Name of the pathPathDesc string*255 Description of the path All this information isstored for every path in the path table of the simulation database(ICASim.mdb).

Simulation Engine

The simulation engine is the interface between the model, the simulationdatabase and the Intelligent Coaching Agent. The simulation engine is ofinterest to the designer so that he can understand the mechanics of itall. But it is the developer of applications using the engine thatshould know the details of the interface (methods & properties) exposedby the engine and the associated algorithms.

Once the designer has constructed the simulation model (ExcelSpreadsheet) and configured all the inputs, outputs & lists, he is readyto test using the test bench included in the ICA Utilities (refer to ICAUtilities documentation). The developer, in turn, needs to implement thecalls to the simulation engine in the GBS application he's building. Thefollowing list identifies the files that need to be included in theVisual Basic project to use the simulation workbench:

wSimEng.cls Simulation Engine class wSimEng.bas Simulation Engine module(this module was introduced only for speed purposes because all the codeshould theoretically be encapsulated in the class) wConst.basIntelligent Coaching Agent constant declaration wDeclare.bas IntelligentCoaching Agent DLL interface wIca.cls Intelligent Coaching Agent classwIca.bas Intelligent Coaching Agent module (this module was introducedonly for speed purposes because all the code should theoretically beencapsulated in the class)

To have a working simulation, a developer places code in differentstrategic areas or stages of the application. There's the Initial stagethat occurs when the form containing the simulation front-end loads.This is when the simulation model is initialized. There's theModification stages that take place when the user makes changes to thefront-end that impacts the simulation model. This is when the ICA isnotified of what's happening. There's the Feedback stage when the userrequests information on the work done so far. This is when thesimulation notifies the ICA of all output changes. Finally, there's theFinal stage when the simulation front-end unloads. This is when thesimulation is saved to disk.

The different stages of creating a simulation, including the VisualBasic code involved, are presented below.

Initial Stage

1. Creating the ICA & the Simulation Engine Objects

Code:

Set moSimEngine=New classSimEngine

Set moICA=New classICA

Description: The first step in using the simulation engine is to createan instance of the class classSimEngine and also an instance of theclass classICA. Note that the engine and ICA should be module levelobject “mo” variables.

2. Loading the Simulation

Code:

1Ret=moSimEngine.OpenSimulation(App.Path & DIR_DATA & FILE_SIMULATION,Me.bookSimulation)

1Ret=moSimEngine.LoadSimulation(mlICATaskID, App.Path & DIR_DATABASE &DB_SIMULATION, 1)

Description: After the object creation, the OpenSimulation andLoadSimulation methods of the simulation engine object must be called.The OpenSimulation method reads the specified Excel 5.0 spreadsheet fileinto a spreadsheet control. The LoadSimulation method opens thesimulation database and loads into memory a list of paths, a list ofinputs, a list of outputs and a list of lists for the specific task.Every method of the simulation engine will return 0 if it completessuccessfully otherwise an appropriate error number is returned.

3. Initializing and Loading the Intelligent Coaching Agent

Code:

1Ret=moICA.Initialize(App.Path & “\” & App.EXEName & “.ini”, App.Path &DIR_DATABASE, App.Path & DIR_ICADOC, App.Path & “\”)

1Ret=moICA.LoadTask(mlICATaskID, ICAStudentStartNew)

Description: The simulation engine only works in conjunction with theICA. The Initialize method of the ICA object reads the application .inifile and sets the Tutor32.dll appropriately. The LoadTask method tellsthe ICA (Tutor32.dll) to load the tut document associated to a specifictask in memory. From that point on, the ICA can receive notifications.

Note: The tut document contains all the element and feedback structureof a task. Ex: SourcePages, SourceItems, TargetPages, Targets, etc . . .

4. Restoring the Simulation

Code:

<<Code to reset the simulation when starting over>>

<<Code to load the controls on the simulation front-end>>

1Ret=moSimEngine.RunInputs(sPaths, True)

1Ret=moSimEngine.RunOutputs(sPaths, True)

1Ret=moSimEngine.RunLists(sPaths, True)

Call moICA.Submit(0)

Call moICA.SetDirtyFlag(0, False)

Description: Restoring the simulation involves many things:

clearing all the inputs and lists when the user is starting over

loading the interface with data from the simulation model

invoking the RunInputs, RunOutputs and RunLists methods of thesimulation engine object in order to bring the ICA to it's originalstate

calling the Submit method of the ICA object with zero as argument totrigger all the rules

calling the SetDirtyFlag of the ICA object with 0 and false as argumentsin order to reset the user's session.

Running inputs involves going through the list of TutorAware inputs andnotifying the ICA of the SourceItemID, TargetID and Attribute value ofevery input.

Running lists involves going through the list of TutorAware lists andnotifying the ICA of the SourceItemID, TargetID and Attribute value ofevery item in every list. The TargetID is unique for every item in alist.

Running outputs involves going through the list of TutorAware outputsand notifying the ICA of the SourceItemID, TargetID and Attribute valueof every output.

Modification Stage

1. Reading Inputs & Outputs

Code:

Dim sDataArray(2) as string

Dim vAttribute as variant

Dim lSourceItemID as long

Dim lTargetID as long

1Ret=moSimEngine.ReadReference(“Distinct _Input”, vAttribute,lSourceItemID, lTargetID, sDataArray)

Description: The ReadReference method of the simulation object willreturn the attribute value of the input or output referenced by name andoptionally retrieve the SourceItemID, TargetID and related data. In thecurrent example, the attribute value, the SourceItemID, the TargetID and3 data cells will be retrieved for the input named Distinct Input.

2. Modifying Distinct Inputs

Code:

Dim vAttribute as variant

Dim lSourceItemID as long

Dim sDataArray(2) as string

vAttribute=9999

sDataArray(0)=“Data Cell #1”

sDataArray(1)=“Data Cell #2”

sDataArray(2)=“Data Cell #3”

1Ret=moSimEngine.WriteReference(“Distinct _Input”, vAttribute,sDataArray)

Description: Modifying a distinct input is as simple as calling theWriteReference method of the simulation object passing the input name,the new attribute value and optionally a data array. The simulationengine takes care of notifying the ICA of the change.

3. Modifying Drag&drop Inputs

Code:

Dim vAttribute as variant

Dim lSourceItemID as long

Dim sDataArray(2) as string

lSourceItemID=1202

vAttribute=9999

sDataArray(0)=“Data Cell #1”

sDataArray(1)=“Data Cell #2”

sDataArray(2)=“Data Cell #3”

1Ret=moSimEngine.WriteReference(“DragDrop _Input”, vAttribute,lSourceItemID, sDataArray)

Description: Modifying a drag&drop input is as simple as calling theWriteReference method of the simulation object passing the input name,the new attribute value, the new SourceItemID and optionally a dataarray. The simulation engine takes care of notifying the ICA of thechange.

4. Reading Lists

Code:

1Ret=moSimEngine.ListRead(sListName, lListIndex, vAttribute,lSourceItemID, lTargetID, sDataArray)

Description: All list in the simulation model can be read one item at atime using the ListRead method of the simulation engine object. Passingthe list name and the index of the item to retrieve, the function willreturn the Attribute value and optionally the SourceItemID, TargetID anddata array of the item specified. Use a looping structure to read entirelists into memory, or to search for and retrieve a particular line item.This will be done quite often as designers generally allow users tomanipulate items from lists. For example, if a user begins to drag anelement of a list, you will need to retrieve this data from the listitem they are dragging.

5. Modifying Lists

Code:

1Ret=moSimEngine.ListAdd(sListName, vAttribute, lSourceItemID,sDataArray)

1Ret=moSimEngine.ListCount(sListName, lTotalItems)

1Ret=moSimEngine.ListModify(sListName, lListIndex, vAttribute,lSourceItemID, sDataArray)

1Ret=moSimEngine.ListDelete(sListName, lListIndex)

Description: The simulation engine object provides basic functionalityto manipulate lists.

The ListAdd method appends an item(SourceItemID, Attribute, Data array)to the list. Let's explain the algorithm. First, we find the top of thelist using the list name. Then, we seek the first blank cell underneaththe top cell. Once the destination is determine, the data is written tothe appropriate cells and the ICA is notified of the change.

The ListCount method returns the number of items in the specified list.The algorithm works exactly like the ListAdd method but returns thetotal number of items instead of inserting another element.

The ListModify method replaces the specified item with the provideddata. Let's explain the algorithm. First, we find the top of the listusing the list name. Second, we calculate the row offset based on theitem number specified. Then, the ICA is notified of the removal of theexisting item. Finally, the data related to the new item is written tothe appropriate cells and the ICA is notified of the change.

The ListDelete method removes the specified item. The algorithm worksexactly like the ListModify method but no new data is added and thecells (width of the list set by ‘Total Columns’) are deleted with the‘move cells up’ parameter set to true. Keep this in mind, as designersoften enter the wrong number of columns in the Total Columns parameter.When they overestimate the Total Columns, ListDelete will modifyportions of the neighboring list, which leads to erratic behavior whenthat list is displayed.

Feedback Stage

1. Running the Simulation

Code:

1Ret=moSimEngine.RunOutputs(sPaths, True)

Description: Inputs and lists notify the ICA when changes happen but notoutputs. Therefore, the RunOutputs method must be invoked beforesubmitting the work for feedback.

2. Triggering the ICA Rule Engine

Code:

1Ret=moICA.Submit(lCoachID)

Description: Once the simulation has been processed, the Submit methodof the ICA object must be called to trigger all the rules and deliverthe feedback. This feedback will be written by the Tutor32.dll to twoRTF formatted files. One file for previous feedback and one file for thecurrent feedback.

3. Displaying ICA Feedback

Code:

Set oViewer=New CFeedbackViewer

oViewer.CoachID=vlCoachID

Call oViewer.DisplayFeedBack(moApp)

Description: The only thing required to display feedback information isto have an RTF control on a form and read-in the feedback files producedby the Submit method of the ICA object.

Final stage

1. Saving the Simulation

Code:

1Ret=moSimEngine.SaveSimulation(App.Path & DIR_DATA & FILE_SIMULATION)

Description: The SaveSimulation method of the simulation engine objectwill save the specified Excel spreadsheet to disk.

SYSTEM DYNAMICS IN ACCORDANCE WITH A PREFERRED EMBODIMENT

To use system dynamics models in the architecture, an engine had to becreated that would translate student work into parameters for thesemodels. A complex system dynamics model to interact with an existingsimulation architecture is discussed below. The system dynamics modelprovides the following capabilities.

1. Allow designers to build and test their system dynamics models andICA feedback before the real interface is built.

2. Reduce the programming complexity of the activities.

3. Centralize the interactions with the system dynamics models.

System Dynamics Engine

As with the simulation engine, the designer models the task that he/shewants a student to accomplish using a Microsoft Excel spreadsheet. Here,however, the designer also creates a system dynamics model (describedlater). The system dynamics engine will read all of the significantcells within the simulation model (Excel) and pass these values to thesystem dynamics model and the ICA. After the system dynamics model runsthe information, the output values are read by the engine and thenpassed to the simulation model and the ICA.

FIG. 47 is a block diagram presenting the detailed architecture of asystem dynamics model in accordance with a preferred embodiment. Oncethe simulation model, system dynamics model and feedback are completelytested by designers, developers can incorporate the spreadsheet in agraphical user interface, e.g., Visual Basic as a development platform.FIG. 47 illustrates that when a student completes an activity, thevalues are passed to the system dynamics engine where the values arethen passed to the system dynamics model (as an input), written to thesimulation model and submitted to the ICA. When the system dynamicsmodel is played, the outputs are pulled by the engine and then passed tothe simulation model and the ICA. Note that the simulation model cananalyze the output from the system dynamics model and pass the resultsof this analysis to the ICA as well. The simulation model can then beread for the output values and used to update on-screen activitycontrols (such as graphs or reports).

It is very important that all modifications that the ICA and systemdynamics model need to know about go through the engine because only theengine knows how to call these objects. This significantly reduces theskill level required from programmers, and greatly reduces the timerequired to program each task. In addition, the end-product is lessprone to bugs, because the model and tutor management will becentralized. If there is a problem, only one section of code needs to bechecked. Finally, since the engine loads the data from the spreadsheet,the chance of data inconsistency between the ICA, the system dynamicsmodel and the application is insignificant.

FIG. 48 is graphical representation of the object model which isutilized to instantiate the system dynamic engine in accordance with apreferred embodiment. The System Dynamics Object Model consists of aspreadsheet object, a system dynamics object, a simulation database,model objects, parameter input objects and parameter output objects. Thefirst object in our discussion is the Spreadsheet object. TheSpreadsheet is the support for all simulation models. The spreadsheetobject is integrated with a Visual Basic development platform inaccordance with a preferred embodiment. The control supports printingand is compatible with Microsoft Excel spreadsheets. With that in mind,designers can use the power of Excel formulas to build the simulation.These spreadsheets can sort or make calculations on the time intervaldata that is received from the system dynamics model, which allows theability to show trends. This functionality allows a large amount of thecalculations and number-crunching to occur in the spreadsheet and not inthe code, placing more control with the activity designers.

The different cells in the spreadsheet model can be configured asparameter inputs or parameter outputs for a system dynamics model. Thisis what the system dynamics engine uses to write and read data from thesystem dynamics model, pass values to the ICA and update the student'swork in the on-line activities. By making the spreadsheet object thecentral repository for the system dynamics data, we ensure that thesystem dynamics model, simulation model, activity and ICA are all insynch.

The system dynamics model generates simulation results over time, basedon relationships between the parameters passed into it and othervariables in the system. A system dynamics object is used to integratewith Visual Basic and the spreadsheet object. The object includes logicthat controls the time periods as well as read and write parameters tothe system dynamics model. With Visual Basic, we can pass theseparameters to and from the model via the values in the simulationobject.

The system dynamics object also controls the execution of the systemdynamics model. What this means is that after all of the parameterinputs are passed to the system dynamics model, the engine can run themodel to get the parameter outputs. The system dynamics object allowsfor the system dynamics models to execute one step at a time, all atonce, or any fixed number of time periods.

When the system dynamics model runs, each step of the parameter inputand parameter output data is written to a ‘backup’ sheet for tworeasons. First, the range of data that is received over time (the modelplaying multiple times) can be used to create trend graphs or used tocalculate statistical values. Second, the system dynamics model can berestarted and this audit trail of data can be transmitted into the modelup to a specific point in time. What this means is that the engine canbe used to play a simulation back in time.

When any event occurs within the system dynamics engine, a log iscreated that tells the designers what values are passed to thesimulation model, system dynamics model and ICA as well as the currenttime and the event that occurred. The log is called “SysDyn.log” and iscreated in the same location as the application using the engine. Aswith the spreadsheet object, the system dynamics object allows a largeamount of the calculations to occur in the system dynamics model and notin the activity code, again placing more control with the activitydesigners.

Model objects are used to configure the system dynamics models withregard to the time periods played. Models are what the parameter inputsand parameter outputs (discussed later) relate to, so these must becreated first. Every model has the following application programminginterface:

Field Name Data Type Description ModelID Long Primary Key for the tableTaskID Long TaskID of the task associated with the model ModelNameString*50 Name of the model (informational purposes) ModelDesc String*50Description of the model (informational purposes) SysDynModel String*50Filename of the actual system dynamics model Start Long Start time toplay modal Stop Long Stop time to play model Step Long Interval at whichto play one model step and record data

This information is stored in the model table of the simulation database(ICASim.mdb). All of the values that will need to be manually entered bythe student that are passed into the system dynamics model areconfigured as parameter inputs (PInputs) objects. Every PInput has aninterface as detailed below.

Field Name Data Type Description PinputID long Primary Key for the tableTaskID long TaskID of the task associated with the parameter inputModelID long ID of the model associated with the parameter inputInputName string*50 Name of the parameter input (informational purposes)InputDesc string*255 Description (informational purposes) ReferenceNamestring*50 Name of the spreadsheet cell associated with the parameterinput¹ SimReferenceName string*50 Name of the associated parameter inthe system dynamics model ¹PowerSim allows designers to createparameters as arrays. If this is the case, then each array item MUSThave one parameter input. What this means is that dynamics arrays cannot be used by the System Dynamics engine.

TutorAware boolean Whether the ICA should be notified of any changes tothe parameter input SourceItemID long SourceItemID of the parameterinput TargetID long TargetID of the parameter input Row long Spreadsheetrow number of the parameter input (used for speed optimization) Columnlong Spreadsheet column number of the parameter input (used for speedoptimization) SheetName string*50 Sheet name were the parameter input islocated (used for speed optimization)

All of this information is stored for every parameter input in thePInput table of the simulation database (ICASim.mdb).

PInputs consist of one spreadsheet cell that can be populated by adesigner at design time or by the GBS application at run time via thesystem dynamics engine object's methods. The purpose of the cell is toprovide an entry point to the simulation and system dynamics models. Anexample of an entry point would be the interest rate parameter in theinterest calculation example. The ICA is notified of any changes to thecell when an appropriate activity transpires. When the ICA is notifiedof a change two messages are sent to the ICA. The first is anICANotifyDestroy message with the parameter input information i.e.,SourceItemID, TargetID and null as an attribute. This message is sent toinform the ICA to remove information from its memory. The second messageis an ICANotifyCreate message with the parameter input information i.e.,SourceItemID, TargetID, Attribute (cell numeric value). This messageadvises the ICA to add this information to its memory.

To demonstrate the use of a parameter input, the interest ratecalculation example is again used as a backdrop to illuminate variousfeatures. FIG. 49 is a PInput Cell for a simulation model in accordancewith a preferred embodiment. FIG. 50 is a PInput backup cell in asimulation model in accordance with a preferred embodiment. InterestRate is the parameter input for this model which will then be used tocalculate balance and interest accumulations. A defined name will alsohave to be created for the backup of the PInput as each time interval isplayed. A requirement for this cell is that it has the same name as theoriginal PInput, but also have the “_BU” extension. The example herewould be “Interest_Rate _BU.” This cell will also have to be created ina column where no other data exists, since all of the backups arewritten below this cell. In the ICA, define a task that will be assignedto the simulation. For example, a TaskID of 123 is generated by the ICA.For this example, we will assume that we want to give feedback on M 5the interest rate selected by the student. In the ICA, define a Targetfor the parameter input.

A PInput table record in accordance with a preferred embodiment ispresented below.

PInputID: 12345 TaskID: 123 ModelID: 1 InputName: Interest Rate inputInputDesc: Interest Rate input into interest calculation modelReferenceName: Interest_Rate SimReferenceName Param_Interest_RateTutorAware: True SourceItemID 1201 TargetID: 4001 Row: 6 Column: 3SheetName: Sheet1

Once the configuration is completed, the designer can also use the ICAUtilities to test the simulation. The Row, Column and SheetName valuesare automatically populated when the designer runs the parameters in theSystem Dynamics Workbench in the ICA Utilities. The reason for obtainingthe cell coordinates is that the processing of cell data issignificantly faster with cell positions than simply using the definedname. The system dynamics engine decodes the defined name (ReferenceName) that the designer entered, and populates the table accordingly.This is an important step because there have been occasions when adesigner would change the layout of a spreadsheet, i.e., move a definedname location, and then forget to perform this step. As such, bizarredata was being passed to the tutor; whatever data happened to reside inthe old row and column. Cells in the spreadsheet that are the outputfrom a system dynamics models can be represented by parameter outputobjects (POutputs). Every POutput has an interface as detailed below.

Field Name Data Type Description PoutputID Long Primary Key for thetable TaskID Long TaskID of the task associated with the parameteroutput ModelID Long ID of the model associated with the parameter outputOutputName String*50 Name of the parameter output (informationalpurposes) OutputDesc String*255 Description (informational purposes)ReferenceName String*50 Name of the spreadsheet cell associated with theparameter output SimReferenceName String*50 Name of the associatedparameter in the system dynamics model TutorAware Boolean Whether theICA should be notified of any changes to the parameter outputSourceItemID Long SourceItemID of the parameter output TargetID LongTargetID of the parameter output Row Long Spreadsheet row number of theparameter output (used for speed optimization) Column Long Spreadsheetcolumn number of the parameter output (used for speed optimization)SheetName String*50 Sheet name were the parameter output is located(used for speed optimization)

All of this information is stored for every output in the Output tableof the simulation database (ICASim.mdb). Each POutput object comprises aspreadsheet cell that is an output from the system dynamics model. Thisis the information that shows the student the results of their choicesand is the main reason for using system dynamics. The POutput can bemade TutorAware, which will notify the ICA of any changes to the cellwhen all the POutputs are processed otherwise the ICA will be unaware ofany changes. When the ICA is notified of a change, two messages are infact sent to the ICA:

1. An ICANotifyDestroy message with the parameter output informationi.e., SourceItemID, TargetID and null as Attribute. This message is toadvise the ICA to remove this information from its memory.

2. An ICANotifyCreate message with the parameter output informationi.e., SourceItemID, TargetID, Attribute (cell numeric value). Thismessage is to advise the ICA to add this information to its memory.

As opposed to PInputs which notify the ICA on every change, POutputs areprocessed in batch just before asking the ICA for feedback.

POutputs use is illuminated below by an example that builds on theprevious interest calculation example. Here, we want to notify the ICAof the balance as it results from changes in the interest rate. FIG. 51is a display illustrating a POutput cell in accordance with a preferredembodiment. The steps required to configure the POutput are presentedbelow.

1. Define a name for cell G4 in Excel. Here we have defined “Balance.”

2. Define the name of the backup cell as “Balance _BU” in a blankcolumn.

3. Let's use the same TaskID as before since the Balance parameter ispart of the same simulation as the Interest Rate parameter. Ex: TaskIDis 123.

4. In the ICA, define a Target for the parameter output. Ex: a TargetIDof 4005 is generated by the ICA.

5. In the ICA, define a SourceItem for the parameter output. Ex: aSourceItemID of 1215 is generated by the ICA.

6. Associate the parameter output to a system dynamics model (refer toModel object discussion).

7. Add the information in the POutput table of the simulation enginedatabase. This configuration can also be done within the ICA Utilities.

The record in the POutput table would look something like this:OutputID: 12347 TaskID: 123 ModelID: 1234 OutputName: Balance ValueOutputDesc: Value of Balance after model has been run ReferenceName:Balance SimReferenceName Param_Balance TutorAware: True SourceItemID1215 TargetID: 4005 Row: 4 Column: 7 SheetName: Sheet1

The following information provides details describing the interactioncomponents in accordance with a preferred embodiment.

Title Description Procedural tasks (w/drag Tasks which require theconstruction of some kind of report drop) with evidence dragged anddropped to justify conclusions Procedural tasks (w/o drag New taskdesigns that are procedural in nature, have very drop) little branching,and always have a correct answer. Ding Dong task Tasks that interruptthe student while working on something else. This template includesinterviewing to determine the problem, and a simple checkbox form todecide how to respond to the situation. Analyze and Decide (ANDIE) Mostcommonly used for static root cause analysis, or task identificationtasks. Developed on SBPC as a result of 3 projects of experienceredesigning for the same skill. Evaluate Options (ADVISE) Used for tasksthat require learner to evaluate how different options meet stated goalsor requirements. Developed at SBPC after 4 projects experienceredesigning for the same skill. Does not allow drag drop as evidence.Run a company task Time based simulation where student “chooses ownadventure”. Each period the student selects from a pre- determined listof actions to take. Developed on SBPC as a simplified version of the BDMmanage task. Use a model task When user needs to interact with aquantitative model to perform what if analysis. May be used for dynamicroot cause analysis - running tests on a part to analyze stress points.ICA Dynamic Meeting Task Developed on BDM to mimic interaction stylesfrom Coach and ILS EPA. Supports dynamic-rule based branching - willscale to support interactions like EnCORE defense meetings and YES.Manage Task Time based simulation where student manages resources. HumanResources Management, managing a budget, manage an FX portfolio. QVIDStatic Meeting Task Developed on Sim2 to support agenda-driven meetingswhere user is presented with up to 5 levels of follow-up questions topursue a line of questioning. As they ask each question, it's follow-upsappear. Flow Chart Task Will support most VISIO diagrams. Developed onSim2 to support simple flow chart decision models. QVID Gather DataStatic flat list of questions to ask when interviewing Componentsomeone. Not used when interviewing skills are being taught (use QVIDStatic meeting task). Supports hierarchical questions and timedtranscripts. Journalize Task Created to support simple journal entrytasks with up to 2 accounts per debit or credit. New Complex Task A newtask that requires a simulation component

Systems Dynamic Engine

The system dynamics engine is the interface between the simulationmodel, the system dynamics model, the simulation database and theIntelligent Coaching Agent. The system dynamics engine is of interest tothe designer so that she can understand the mechanics of it. Once thedesigner has constructed the simulation model (Excel Spreadsheet), builtthe system dynamics model (PowerSim) and configured all of the parameterinputs and parameter outputs, a test can be performed using theworkbench included in the ICA Utilities (refer to ICA Utilitiesdocumentation). The developers, in turn, need to implement the calls tothe system dynamics engine in the GBS application that is being built.The following list identifies the files that need to be included in theVisual Basic project to use the system dynamics engine.

WSysDynEng.cls System dynamics Engine Class wSysDynEng.bas Systemdynamics Engine module (this module was introduced only for speedpurposes because all the code should theoretically be encapsulated inthe class) wConst.bas Intelligent Coaching Agent constant declarationwDeclare.bas Intelligent Coaching Agent DLL interface wIca.clsIntelligent Coaching Agent Class wIca.bas Intelligent Coaching Agentmodule (this module was introduced only for speed purposes because allof the code should theoretically be encapsulated in the class)

To utilize the system dynamics engine fully, the developer must placecode in different strategic areas or stages of the application.

1) Initial stage—the loading of the form containing the simulationfront-end. This is when the simulation model and system dynamic engineare initialized.

2) Modification stage—Takes place when the user makes changes to thefront-end that impacts the simulation model PInputs). This is when theICA is notified of what's happening.

3) Run stage—The system dynamics model is run and parameter outputs arereceived.

4) Feedback stage—The user requests feedback on the work that they haveperformed. This is when the simulation notifies the ICA of all outputchanges.

5) Final stage—The simulation front-end unloads. This is when thesimulation model is saved.

These stages will be explained by including the Visual Basic codeinvolved as well as a short description of that code.

Initial Stage Code in Accordance With a Preferred Embodiment

1. Creating the ICA & the Simulation Engine Objects

Code:

Set moSysDynEngine=New classSysDynEngine

Set moICA=New classICA

Description: The first step in using the system dynamics engine is tocreate an instance of the classSysDynEngine class and also an instanceof the classICA class. Note that the engine and ICA should be modulelevel object “mo” variables.

2. Loading the Simulation

Code:

1Ret=moSysDynEngine.OpenSimulation(FILE_SIM, Me.bookSim, True)

1Ret=moSysDynEngine.LoadSysDyn(mlICATaskID, DB_SIMULATION, 1)

1Ret=moSysDynEngine.LoadModel(MODEL_NAME,mbTaskStarted)

Description: After the object creation, the OpenSimulation,LoadSimulation and LoadModel methods of the system dynamics engineobject must be called. The OpenSimulation method reads the specifiedExcel 5.0 spreadsheet file (FILE_SIM) into a spreadsheet control(bookSim). The LoadSysDyn method opens the simulation database(DB_SIMULATION) and loads into memory a list of parameter inputs and alist of parameter outputs. The LoadModel method opens a system dynamicsmodel (MODEL_NAME). Every method of the system dynamics engine willreturn 0 if it completes successfully otherwise an appropriate errornumber is returned.

3. Initializing and Loading the Intelligent Coaching Agent

Code:

1Ret=moICA.Initialize(App.Path & “\” & App.EXEName & “.ini”, App.Path &

DIR_DATABASE, App.Path & DIR_ICADOC, App.Path & “\”)

1Ret=moICA.LoadTask(mlICATaskID, ICAStudentStartNew)

Description: The system dynamics engine only works in conjunction withthe ICA. The Initialize method of the ICA object reads the application.ini file and sets the Tutor32.dll appropriately. The LoadTask methodtells the ICA (Tutor32.dll) to load the tut document associated to aspecific task in memory. From that point on, the ICA can receivenotifications. Note: The tut document contains all the element andfeedback structure of a task. Ex: SourcePages, SourceItems, TargetPages,Targets, etc . . .

4. Restoring the Simulation

Code:

1Ret=moSysDynEngine.RunPInputs(MODEL_NAME, True)

1Ret=moSysDynEngine.RunPOutputs(MODEL_NAME, True)

1Ret=moSysDynEngine.PassPInputsAll

Call moICA.Submit(0)

Call moICA.SetDirtyFlag(0, False)

Description: Restoring the simulation involves many things:

clearing all of the parameter inputs and outputs when the user isstarting over

loading the interface with data from the simulation model

invoking the PassPInputsAll method of the system dynamics engine objectin order to bring the ICA to its original state

invoking the RunPInputs and RunPOutputs methods of the system dynamicsengine object in order to bring the system dynamics model to it'soriginal state

calling the Submit method of the ICA object to trigger the ICA to playall of the rules

calling the SetDirtyFlag of the ICA object to reset the user's session.

Running parameters involves going through the list of TutorAware PInputsand POutputs and notifying the ICA of the SourceItemID, TargetID andAttribute value of every one.

Modification Stage

1. Reading Parameter Inputs & Outputs

Code:

Dim sDataArray(2) as string

Dim vAttribute as variant

Dim lSourceItemID as long, lTargetID as long

1Ret=moSysDynEngine.ReadReference(“Input _Name”, vAttribute,lSourceItemID, lTargetID, sDataArray)

Description: The ReadReference method of the system dynamics object willreturn the attribute value of the parameter input or output referencedby name and optionally retrieve the SourceItemID, TargetID and relateddata. In the current example, the attribute value, the SourceItemID, theTargetID and 3 data cells will be retrieved for the parameter inputnamed Input _Name.

2. Modifying Parameter Inputs

Code:

Dim vAttribute as variant

Dim lSourceItemID as long

Dim sDataArray(2) as string

vAttribute=9999

sDataArray(0)=“Data Cell #1”

sDataArray(1)=“Data Cell #2”

sDataArray(2)=“Data Cell #3”

1Ret=moSysDynEngine.WriteReference(“Input _Name”, vAttribute,sDataArray)

Description: To modify a parameter input, call the WriteReference methodof the system dynamics object and pass the PInput reference name, thenew attribute value and optionally a data array (an additionalinformation to store in the simulation model). The system dynamicsengine notifies the ICA of the change.

Run Stage

1. Playing the System Dynamics Model

Code:

1Ret=moSysDynEngine.PlayModel(SYSDYN_PLAYSTEP)

lblCurrentTime.Caption moSysDynEngine.CurrentTime

lblLastTime.Caption=moSysDynEngine.LastTime

Description: Playing the system dynamics model is also handled by thesystem dynamics engine. There are three ways that the models can beplayed, an at once, one step at a time (shown above) or until a specificpoint in time. These are the parameters that are passed into thePlayModel method. Playing of the model generates the parameter outputvalues and passes the Tutor Aware POutputs to the ICAT. The engine alsokeeps track of time and these values can be read using the CurrentTimeand LastTime properties.

2. Jumping Back in a System Dynamics Model

Code:

1Ret=moICA.LoadTask(mlICATaskID, ICAStudentStartNew)

1Ret=moSysDynEngine.JumpBack(TIME_TO_JUMP_TO)

Description: Because the system dynamics engine writes backup copies ofthe parameters passed to and from it, it can start over and resubmitthese values back to the system dynamics model until a given period oftime. To do this, the code would need to restart the ICA and then callthe system dynamics engine to jump back to a given time(TIME_TO_JUMP_TO).

Feedback Stage

1. Triggering the ICA Rule Engine

Code:

1Ret=moICA.Submit(lCoachID)

Description: Once the simulation has been processed, the Submit methodof the ICA object must be called to trigger all the rules and deliverthe feedback. This feedback will be written by the Tutor32.dll to twoRTF formatted files. One file for previous feedback and one file for thecurrent feedback.

2. Displaying ICA Feedback

Code:

Set oViewer=New CFeedbackViewer

oViewer.CoachID=vlCoachID

Call oViewer.DisplayFeedBack(moApp)

Description: The only thing required to display feedback information isto have an RTF control on a form and read-in the feedback files producedby the Submit method of the ICA object.

Final stage

1. Saving the Simulation Model

Code:

1Ret=moSysDynEngine.SaveSimulation(FILE_SIMULATION)

Description: The SaveSimulation method of the system dynamics enginewill save the specified Excel spreadsheet to disk.

Source Items and Targets relate to specific on-line objects. When theseobjects are selected in an activity, an associated Source Item andTarget are ‘mapped’ into an ICA, which then looks at all of theconfigured rules and displays an appropriate feedback (known in the ICAas a Coach Item). For example, if an activity required users to drag anaccount name (Source Item) to a Journal Entry (Target), the ICA would benotified of this association and feedback would be delivered based on aset of predefined rules.

Feedback (Coach Items) can be displayed in two ways, as a parent or as achild. Parent feedback can be Stand Alone text where it is the onlypiece of feedback delivered, or it can be used as a header which willsupport many ‘children’ pieces of feedback. An example of a Parentheader would be feedback that stated “Look at your Journal Entries, hereis what I see . . . ” Below this would be multiple line items thatrelate to specific feedback given to the student about a Journal Entry.

This structure will be used for the on-line meetings as well. Instead ofthe association of Source Items and Targets occurring when an item isdragged, it occurs when a question is selected by the student. Ruleswill be configured based on these mappings to fire specific feedback.The Parent header, instead of being text, will include video informationsuch as the video to be played. The children feedback includes allassociated follow-up questions.

ICA Configuration in Accordance With a Preferred Embodiment

FIG. 52 is an overview diagram of the logic utilized for initialconfiguration in accordance with a preferred embodiment. Since thestructure of the feedback is the same as other on-line activities, theICA can also be configured in the same manner. For ease of creation andmaintenance of ICA feedback, it is recommended that the feedback isconstructed so that only one rule fires at any point in time. Note thatthe organization of the example is one of many ways to structure thefeedback.

Step 1: Create a Map of Questions and Follow-up Questions

Before designers start configuring the ICA, they should draw a map ofthe questions, videos and follow-up questions that they wish to use inthe on-line meeting. This will give them a good understanding of theinteractions as they configure the ICA.

Step 2: Create a Coach

All feedback is given by a coach. Create a specific coach for theon-line meeting.

Step 3: Create the Source Items and Targets

FIG. 53 is a display of the source item and target configuration inaccordance with a preferred embodiment. Every question will have oneSource Item (1) and Target (2) associated with it. These will be used bythe ICA to show videos and follow-up questions. For organizationalpurposes and ease of reading, it is recommended that each Source Page(“0 Intro”) contain all of the follow up questions (“Intro Q1”, “IntroQ2”, “Intro Q3”). Targets can be created one per Source Item (shownhere) or one per many Source Items. This is not very important, so longas there are distinct Source Item and Target associations. Once theSource Items and Targets have been created, associate them intoSourceItemTargets (3) and give them a relevance of one. These are theunique identifiers which the ICA will use to fire rules and to providefeedback to the student.

Step 4: Create the Parent Header (Video Information)

FIG. 54 is a display of video information in accordance with a preferredembodiment.

Feedback (Coach Items) are organized into Target Groups (1). In FIG. 54,each on-line question has one Target Group for ease of maintenance. EachTargetGroup must have at least one related Target (4). These are theSourceItemTarget mappings that were made at the end of Step 3. Next,Rules (2) are created to fire when the SourceItemTarget is mapped (aquestion is clicked). Coach Items (3) are associated to a rule andrepresent the feedback which will be shown if the rule is fired.

FIG. 55 illustrates a display depicting configured rules in accordancewith a preferred embodiment. Rules are configured to fire when specificSource Items and Targets are mapped (when a user clicks on a question).For this reason, Aggregate Rules are configured that only look to see ifthis mapping has occurred. To have the rules query these mappings, theTarget Group field (1) is equated to the Target that was mapped to thisTarget Group. For the rule to fire, special criteria have to besatisfied. The Source Item and Target are assigned a relevance of one sothey will be recognized as a correct mapping (or UCP). Therefore, thisrule fires if there is a minimum of one correct mapping, or UCP (2).Using this format, only one rule will fire at any point in time becauseonly one question will be selected at any point in time.

FIG. 56 illustrates feedback for configured rules in accordance with apreferred embodiment. Each rule has associated feedback (Coach Items)that depict when a rule is fired. To configure this feedback as aheader, this Coach Item must be configured as a parent (1). Since thisCoach Item is a header and will show other children feedback, the numberof children displayed must also be set (2). This will be the number offollow up questions for the selected question. The feedback window iswhere the header text is configured relating the video information thatwill appear as a result of a question being selected (the SourceItem andTarget mapping).

To separate the video information, the feedback text includes specifictags. To state the filename for the video played, the name must beinside the <F> and </F> tags. The start time for the video to play usesthe <I> and </I> tags and the stop time uses the <O> and </O> tags.Transcripts can also be used to show on screen or for the purposes oftesting feedback without video. The tags for transcripts are <T> and</T>.

Step 5: Create the Children (Follow-up Questions)

FIG. 57 illustrates a display with follow-up configuration questions inaccordance with a preferred embodiment. To configure the follow-upquestions, each follow-up question is defined as a child in the sametarget group as the header. Remember that the header here was configuredto have three children and there are also three follow-up questionchildren configured. Each child also has one Rule and Coach Itemassociated with it.

FIG. 58 illustrates configuration of aggregate rules in accordance witha preferred embodiment. The Aggregate Rules for the children areconfigured exactly the same as the parent header. Notice that the TargetGroup Target is the same Target as the parent. The Rule is also firingwhen this Target Group has a positive mapping (UCP of one). These rulesare created in the same way so that the parents and children all fire atthe same time.

FIG. 59 illustrates a set of coach items in accordance with a preferredembodiment.

The Coach Items for the children represent the follow-up questions. Thecoach items must be configured as children (1) so that they are properlyassociated with their respective parent. The feedback text (2) is thecaption for the follow-up question.

Configuring Utilities

Once the ICA configuration is complete, there is one last step to theprocess. In order for the selection of a question to drive otherquestions and videos, each question must relate to one Source Item andone Target. This way, when any question is selected, the ICA is notifiedand the next video and group of follow-up questions can be displayed. Inthe ICA Utilities Suite, in accordance with a preferred embodiment,there is an ICAMeeting Configuration tool which maps the individualCoach Items (Questions) to a Source Item and a Target. The Coach Item IDto be entered is the question that is selected by the user and theSource Item and Targets entered relate to the Target Group Targets thatdrive the video and follow up questions.

FIG. 60 is an ICA Meeting Configuration tool display in accordance witha preferred embodiment. To add a new association, click on the Add Newbutton on the toolbar (1). Here, designers can type the Coach Item,Source Item or Target Ids to associate. Another utility, the ObjectViewer, can be used, which will display all of the relevant Coach Items,Source Items and Targets. These can then be dragged to the respectivefields. All of the associations can be viewed from the grid depicted onthe left side of the utility (2) in FIG. 60.

Using the ICAMeeting in Visual Basic

Once the ICAMeeting has been configured, it can be implemented or testedusing Visual Basic. This would represent the on-line questions andvideos that are driven by the ICA feedback. Below are the steps requiredto perform this action. In order to use the ICAMeeting in Visual Basic,the xICAMeeting.cls and xICAMeeting.bas files are required. Note thatthe Visual Basic components required for the ICA (wICA.cls, wICA.bas,wConst.bas, wDeclare.bas) are also required for the ICAMeeting class towork.

Step 1: Create the Controls Needed for the ICA Meeting

Create a command button as a control array for the questions

Create a picturebox for the video to play

Create a RichTextbox control to receive the ICA feedback

Create a textbox for the transcripts of the video to appear

Step 2: Configure the ICA Meeting

Initialize class

Set moICAMeeting=New classICAMeeting

Configure parameters:

Set coachID to the ID created in the ICA for the coach

moICAMeeting.CoachID=4

State if videos should show the control box to play and stop videos

moICAMeeting.ShowClip=True

Initialize class and pass in Question Button, Rich text control, Videopicturebox and Transcript text field

Call moICAMeeting.Initialize(cmdQuestion( ), rtxtHeader, picVideo,txtTranscript)

Set Question Click Event and pass in index of control array buttonclicked Call moICAMeeting. OnQuestionClick(Index)

Set Restart method (if desired) and pass in the ID of the task asconfigured in the ICA

Call moICAMeeting.RestartMeeting(mlICATaskID)

Debugging

When debugging the on-line meeting, check that the followingrequirements exist. If any of these criteria are not met, the meetingwill not work properly.

Target Groups

Target Groups

Must have a Target that relates to a Source Item and Target Mapping ()

Should contain the header and a few children

Parent Coach Items (Video Information)

Rules

Must use the coach defined for the activity

Aggregate Rule

Must have the Target that was assigned to the Target Group ()

Must have a UCP minimum of 1

Coach Items

Must be designated as a parent

Must contain at least one child

Feedback must be configured using the <F>,<I>,<O> and <I> tags

Children Coach Items (Follow Up Questions)

Rules

Must use the coach defined for the activity

Aggregate Rule

Must have the Target that was assigned to the Target Group ()

Must have a UCP minimum of 1

Coach Items

Must be designated as a child

Feedback must include text for a follow up question

Intelligent Coaching Agent (ICA) Utilities

The Intelligent Coaching Agent Tool (also known as the tutor) was usedto create remediation for the activities within the course and thearchitecture passed values to the tutor. One drawback was that thearchitecture did all of the processing and, therefore, all of thesimulation. This was not the most data driven or most efficient way ofcreating business simulation because any changes to activities had tomade within code.

The ICA Utilities incorporate business simulation into a multimediaapplication. What this means is that there is now a middle layer betweenthe application and the ICAT. These utilities, along with the simulationengine (described later), allow the architecture to be a front end tothe simulation. Now, any changes to a simulation model do not need to beincorporated into code. The ICA Utilities and simulation engine workwith simulation models created in Microsoft Excel. After the model iscreated, the designer uses the Defined Name function in Excel to flagspecific cells that are to be used by the application and the ICAUtilities in accordance with a preferred embodiment. FIG. 62 illustratesan ICA utility in accordance with a preferred embodiment. The ICAUtilities consist of six utilities that work with the IntelligentCoaching Agent Tool (ICAT) to incorporate business simulation with themultimedia application. Below is a description of each utility, whichwill be discussed in detail following this introduction.

The Object Editor is used for the configuration of objects thattranslate simulation variables into values passed to the ICAT. This isreally where the “middle layer” of the simulation is configured.

The Simulation Workbench allows designers to test their spreadsheetsonce they have configured the simulation. Therefore, the testing offeedback can start well before testing, or even before any code iswritten at all!

The Object Viewer is a tool that shows the designer the ICAT objects.This can be used for viewing purposes without using the ICAT.

The Log Viewer shows all of the logs associated with the ICAT. This ishelpful in debugging feedback received in the Simulation Workbench.

The ICA Doc Maker also designers to create TutorDoc files. These are thefinal outputs of the ICAT, which are used by the application toremediate.

The Feedback Reviewer utility allows designers to resubmit previouslysubmitted work to the ICAT.

Navigation:

FIG. 62 illustrates a configuration utility display in accordance with apreferred embodiment. When first entering the Utilities, a user mustselect their user name (1) and the Task they wish to work on (2). Usernames can be added in the Object Editor (discussed later). Some of theutilities require user names to be selected and will not open withoutthem. To open any of the ICA Utilities, users select the utility from atoolbar (3), or use the Utilities menu item which is accessible from anyscreen. Depending on which utility is open, other menu options becomeavailable. Because the ICA Utilities have six different utilities thatcan be opened at one time, these windows can be arranged for ease inviewing. The Window menu item, which is accessible from any screenallows multiple windows to be cascaded, tiled horizontally or tiledvertically.

At the bottom of the ICA Utilities, there is a status bar that relaysinformation to the user. When the mouse is moved over key items in theutilities, such as the toolbar icons or utility buttons a description ofwhat these objects do appears on this status bar. The status bar alsodisplays information when processing is occurring as to what the utilityis currently doing.

The Object Editor:

The Object Editor is used to translate application information intovalues for the ICAT, which can then be remediated upon.

FIG. 63 illustrates an object editor toolbar in accordance with apreferred embodiment. The Object Editor uses this toolbar on the side ofeach configuration display. To add a new object, the Add New button isselected. To edit an existing object, highlight that object and click onthe Edit button. To delete an existing object, highlight the object andclick the Delete button. When an object is being added or edited, the OKand Cancel buttons become enabled. To save changes, the OK button isselected and to cancel any changes, the Cancel button is selected.Objects are scrolled by using the arrow buttons on the bottom of thetoolbar. There is also a counter that displays the current record andhow many total records are currently defined.

FIG. 64 illustrates the seven areas that can be configured for asimulation in accordance with a preferred embodiment.

Paths are used to pass select information to the ICAT. If specific dataneeds to be passed to one coach (the ICAT allows for multiple teammembers to give feedback), while other data needs to be passed to adifferent coach, two Paths can be used to allow all of the data to bestored in one simulation model.

FIG. 64 illustrates a display that defines inputs in accordance with apreferred embodiment.

Inputs are configured for the contributions in a simulation model. Usinga model of a+b=c, “a” and “b” would be inputs. To configure an input, aname and description are given for informational purposes. A referencemust also be provided. This is the Defined Name on the simulationspreadsheet where the input value resides. This reference is used by theSimulation Engine to locate the sheet and cell location of the input.Note that the Simulation Workbench can configure and view these definednames. These defined names can be typed in or dragged from theSimulation Workbench utility. A path must also be selected for an input.This is where a designer can be selective as to what information to passto a coach in the ICAT. Because of this, at least one path must becreated before an input can be properly configured.

Inputs can also be used by the application, but not passed to the ICAT.To pass objects to the ICAT, a designer must specify the awareness ofthe Input tutor of the input. If the Input is to be passed to the ICAT,then a TargetID must be given to this input. Here is where the ObjectViewer can be used. Target Ids can be typed in or dragged from theObject Viewer. SourceItemIDs can also be configured here. This shouldonly be done if the Input has only one choice (such as a textbox).Multiple choices, such as a combobox or option buttons, allow formultiple SourceItemIDs and therefore, in those cases, this field shouldbe left blank. Outputs are configured for outputs in the simulationmodel. Using the same example as above (a+b=c), “c” would be the output.Outputs are derived from inputs into a model. Outputs are configuredexactly the same as inputs.

FIG. 66 illustrates a list editor in accordance with a preferredembodiment.

Lists are used to pass multiple objects to the ICAT. This is useful whenthere are many items to be passed to the tutor that are not static. Forexample, a drag-drop area where any number of items can be dragged overcan be configured as a List. Dragging points over would add to the list,and dragging points off would delete from the list (and the ICAT). Toconfigure a list, the designer must use multiple columns in thesimulation model and no other information can be used in these columns.This is because when a list deletes an item, it shifts up all othercells below it. The defined name for the list is the first row where thefirst value resides. Lists also use the Name, Description, Reference andPath fields. Note that lists can also be Tutor Aware and must beassigned to a target. The one field used by a list that is differentthan an input or an output is the Total Columns field. This processdefines how many columns are used by the list, including the definedname of the list.

Students are configured for the ICA Utilities. FIG. 67A illustrates adefine student display in accordance with a preferred embodiment.Students are the designers of the simulation models. A student must beselected before the other utilities can be used. Therefore, addingstudents should be the first task when using the utilities. Student nameand description are used for informational purposes. The student ID isan identifier for the user and can be any number.

ControlSourceItems are SourceItemID values that can be stored to be usedby the application. FIG. 67B illustrates a ControlSourceItem display inaccordance with a preferred embodiment. SourceItemIDs are Ids that theapplication must pass to the simulation engine, which then passes themto the ICAT. A SourceItemID relates to one data object that is beingremediated on, such as a text field of account number. UsingControlSourceItems, the SourceItemIDs no longer have to stay hard-codedin the application and can change without any effects on code.ControlSourceItems can be configured for a combobox of all twelvemonths. Therefore, the first item in the combobox can be January, thesecond can be February and so on.

When the user selects a month, the application uses the index of thecombobox to find the ControlSourceItem and pass that to the simulationengine.

ControlSourceItems are configured using a name and description forinformational purposes. Module Name refers to the task that these itemsreside in. These can be used for logical groupings ofControlSourceItems. The Item number is an index used to distinguishbetween ControlSourceItems (for example, the combobox listindexproperty). The SourceItemID for that ControlSourceItem is also neededand can be dragged from the object editor.

ControlTargets are like ControlSourceItemIDs, but instead of storingSourceItemIDs they store TargetIDs. If a SourceItem is something that isdragged from, then a Target is something that is dragged to.

The Simulation Workbench:

The Simulation Workbench is used by designers to test the feedbackcreated in the ICAT. It can also be used to configure simulation models.Simulation models can be imported by using the File menu path and thenOpen. FIG. 68 illustrates a simulation workbench in accordance with apreferred embodiment. Once a simulation model has been loaded, thedesigner can enter values into their inputs and outputs and test thefeedback. Notice here that the example of 1+2=3 is used with 1 and 2being configured as inputs and 3 an output.

When a cell with a defined name is highlighted (here it is call B6), theDefined Name appears in the Active Cell Name field (1). This definedname can be dragged from this field to the Object Editor forconfiguration purposes. To run a simulation, the utilities need to bestarted. Click on the Start Over button (2). At this time, all of thePaths associated with that task will populate the Path list (3). Also,any coaches configured in the ICAT will populate as buttons on thebottom of the toolbar (5) with an associated path. To run a simulation,select the simulation and click on the Run Simulation button (4). Byrunning the simulation, all of the defined inputs, outputs and lists arepassed to the simulation engine which then passes the TutorAware objectsto the ICAT. The remediation can now be viewed by clicking on any of thecoaches on the bottom of the toolbar (5). By utilizing a SimulationWorkbench, a designer can change inputs and outputs to simulate what theapplication will do and see their feedback, without any code beingwritten yet.

The Object Viewer:

The Object Viewer is a snapshot of the ICAT configuration. Although ICATobjects, such as Targets and SourceItems cannot be configured in theobject viewer, the utility is good for viewing the objects as feedbackand is used in the Simulation Workbench. FIG. 69 illustrates an objectviewer in accordance with a preferred embodiment. As shown in FIG. 69,the object viewer lists the SourcePages, Target Pages and Target Groupsfor a selected task. By examining further details associated with theseobjects, designers can obtain specific information, such as SourceItemIDnumbers and the values that are mapped as correct answers. SourceItemIDsand TargetIDs can be dragged from the graphical hierarchy on the left tothe Object Editor to configure Inputs, Outputs, Lists,ControlSourceItems and ControlTargets.

FIG. 70 illustrates an Object Viewer Configuration in an Utilities menuin accordance with a preferred embodiment. The object viewerconfiguration display facilitates interactive user selection of ICATobjects to view in the Object Viewer. These selections are saved for thedesigner as their preferences so that the next time the user utilizes autility, the preferences are utilized as the user's predefined settings.

The Log Viewer:

The Log Viewer utility is used to view the logs created by the ICAT.These are very helpful in debugging feedback. FIG. 71 illustrates a logviewer in accordance with a preferred embodiment.

The Debug Log shows every object passed to the ICAT. If an account wasdragged to a journal page, then the SourceItemID (account) and target(Journal page) are mapped with the attribute (amount journalized). If anobject is deleted, it is also noted here.

The General Log shows general ICAT data such as the Target Groups, Rulesand feedback received.

The Load Log shows the ICAT objects used when the ICAT was loaded.

The Student Log groups ICAT data by Target Group and shows the number ofcorrect, incorrect or extra items in that group. This log also showsevery ICAT rule as well which ones have been fired and which ones havenot.

The Last Submission Log shows the feedback received from the lastsubmission to ICAT.

The Error Log shows any errors that were incurred by the ICAT.

The Doc Maker:

The Doc maker is used to make ICA Docs, which are used by theapplication and the Simulation Workbench to process information and giveremediation. FIG. 72 illustrates a Doc Maker display in accordance witha preferred embodiment. To create an ICA Doc, a user selects thedatabase from where the ICAT data is stored. Then, select the DocumentPath where the ICA Doc will be created to. Finally, select the desiredtasks and click on the Make Docs button.

The Feedback Reviewer:

The feedback reviewer utility is used after the configuration process iscomplete and other users are working with the application. Theapplication stores all of the ICAT submissions in a student table, whichcan then be passed back to the ICAT after changes have been made. FIG.73 illustrates a Feedback Reviewer display in accordance with apreferred embodiment. A user first selects a saved student profile bypositioning the cursor over and clicking the Student combobox (1). Thisaction invokes logic which then populates any tasks that the studentperformed in the Task list (2). By selecting a task, all of thesubmissions that the student performed populate the submission table(7).

To view a submission, click on the submission in the submission table(7). This will populate all of the Targets, SourceItems and Attributessubmitted at that time in the submission data table (6). Also, anycomments added by the tester in the application will appear in theTester Comment Field (8) as well as the feedback received for thatsubmission (9). To resubmit this data to the ICAT, click on the LoadArchive button (3). This action loads the SourceItems, Targets andAttributes from the Submission Data (6) into the ICAT. Then, this datacan be replayed one step at a time by clicking the Replay button (5) orall of the data for all submissions can be replayed by clicking on theReplay All button (4). After this data is replayed, the Current Feedbackfield (11) is populated with the feedback received. Any comments can beadded to the Fixer Comments field (10). This utility efficientlyfacilitates student submissions transmission to the ICAT withoutrecreating the work. ICAT rules can be configured and then thesubmissions can be replayed to test the associated changes.

Example in Accordance With A Preferred Embodiment

The following example is provided to step through the process for usingthe ICA Utilities:

Objective:

The objective here is to create a task where users will journalize aninvoice and receive feedback on their work.

Step 1) Configure the ICAT:

After planning the task, the designer should add all relevantinformation to the ICAT such as the SourceItems (Accounts), Targets(Invoices), Attributes (Amounts to Journalize) and any Rules they wishto create. For this example, the correct answer is created in the ICAT(Debit Machinery for $1,000 and credit Accounts Payable for $1,000)along with some basic rules and feedback.

Step 2) Create the Simulation Model:

The tables below represent the model for the example simulation.

Accounts SourceItem Accounts Payable 1 Accounts Receivable 2 Cash 3Machinery 4 Account SID Amount Debit DR_AMOUNT Credit CR_AMOUNT Invoice1 Wills Machinery Two pressing machines were purchased on account for$1,000.

The three tabular displays appearing above show an invoice associatedwith the purchase of two machines on account. We also see theSourceItemIDs for the possible accounts (these were configured in theICAT). In the simulation model, defined names were given for the Amountfields in both the Debit (DR_AMOUNT) and Credit (CR_AMOUNT) fields. TheSourceItemID field is created to the left of the attribute field and theattribute field always has the defined name. This is because thesimulation engine finds the Defined Name and gets the attribute fromthere. Then, it looks to the left of the defined name to find theSourceItemID.

Step 3) Configure the Inputs, Outputs and Lists

For this example, only 2 inputs are needed and they are the debit andcredit entry for the invoice. In the Object editor, create a path to beused to pass the inputs to the ICAT. Then, configure the inputs usingthe DR_AMOUNT and CR_AMOUNT defined names and the Target defined in theICAT. FIG. 74 is an object editor display that illustrates the use ofreferences in accordance with a preferred embodiment. The reference isused in the defined name (DR_AMOUNT), the Input is Tutor aware and willbe mapped to TargetID 300 (created in the ICAT to distinguish the debitfor this invoice). The credit input is created in the same way.

Step 4) Test the Feedback in the Simulation Workbench

Designers can open the Simulation Workbench and load the model that wascreated in Step 2. Then, different SourceItemIDs for the accounts andthe amounts can be changed in the model. a During this time, designerscan Load and Run the Simulation to see the feedback. One example entailsthe step of putting the Machinery SourceItemID (4) in the Debit SIDfield, 1,000 in the Debit Amount field, Accounts Payable SourceItemID(1) in the Credit SID field and 1,000 in the Credit Amount field to seeif they get praise by the Coach.

Step 5) View and Debug Errors

After submitting multiple times to the ICAT, a designer can view whatwas passed to the tutor by viewing the logs in the log viewer. If therewas an error, such as the correct answers being put in but incorrectfeedback showing, these logs would prove helpful in tracking down theproblem. Designers can also look in the Object Viewer to see the actualICAT configuration.

The combination of the Log Viewer and ICAT Viewer will help the designerin testing and finding any problems in their feedback.

Step 6) Making Changes are Fixing Errors

Once the problems have been tracked down (Step 5), a designer can makethe appropriate changes in the ICAT. From the ICA Doc Maker utility, anew ICA Doc can be made and then retested all over again.

Step 7) Building the Task

After the task has been designed and feedback created, the coder can usethe ControlSourceItem object in the Object Editor utility to map theSourceItemIDs to specific accounts. Therefore, when a user drags anaccount from the chart of accounts, the application retrieves thatSourceItemID from the ControlSourceItem list and then passes it to theSimulation Model.

FIG. 75 presents the detailed design of smart spreadsheets in accordancewith a preferred embodiment. Processing commences at function block 7500where the excel spreadsheet is designed to model to perform scenarioplanning for the application that the business simulation is targetedfor. By way of example, a model for real estate that analyzes an ownversus rent decision is utilized to convey features in accordance with apreferred embodiment. Function block 7510 illustrates the next stepwhich entails associating drivers for specific analysis tasks that areused in the model. For example, the price of unit, down payment, taxrate, estimated appreciation, assessment, rent, annual rent increase,type of loan, and salary will each be utilized in evaluating anformulating the decision. Then, at function block 7520, a loanamortization schedule is created to track the ten year equity growth,tax savings, portfolio value, net gain/loss schedules.

The next step entails designing the tutor approach. First, at functionblock 7530, the expert metrics are identified for home buying metrics.These include the ratio of a person's salary to their home loanpayment+assessment, new payment/rent, five year gain, % down, scenarioassumptions regarding market and real estate appreciation. Then, atfunction block 7540 the relative weights for each metric are establishedand the rule structures are established that identify an appropriateconclusion to reach. For example, praise would entail a message sayinghome is a good buy, polish would entail a message that the home may be agood buy, but several risks should be addressed, focus parent wouldentail a message that the home is not a good buy due to the followingindicators, and list the indicators suggesting that the home is not agood buy. Finally, a redirect message would be: are you kidding, theinputs are entirely unrealistic.

Function block 7550 creates the focus child feedback based on aprioritization of key metrics such as the break even is too long, andthe appreciation isn't high enough to justify the estimated foregonestock market appreciation, or there is not enough money down to growequity in a short period of time. Finally, as function block 7560suggests, the feedback is tested with sample scenario data and a usertest model is created to capture user questions at interaction points ofrelevance, questions are attached to the tutor regression database, andthe feedback is fixed and tested in the regression workbench.

While various embodiments have been described above, it should beunderstood that they have been presented by way of example only, and notlimitation. Thus, the breadth and scope of a preferred embodiment shouldnot be limited by any of the above described exemplary embodiments, butshould be defined only in accordance with the following claims and theirequivalents.

What is claimed is:
 1. A method for creating a business simulation thatprovides remediation during a dynamic, goal based educational learningexperience, comprising the steps of: (a) presenting information thatmotivates accomplishment of a goal for use in the business simulation;(b) monitoring, by a rule based expert system, answers to questionsposed to evaluate progress of a student toward the goal; and (c)providing dynamic, goal-based remediation learning information feedbackusing a remediation object component, the remediation object componentbeing a component of the rule based expert system, the remediationobject component including a knowledge base that generatesindividualized coaching messages based on the answers that furthermotivate accomplishment of the goal.
 2. A method for creating a businesssimulation as recited in claim 1, wherein the remediation objectcomponent further includes a knowledge system and a software tutor.
 3. Amethod for creating a business simulation as recited in claim 1, andfurther comprising the step of determining a profile of the student,wherein the artificial intelligence engine generates the individualizedcoaching messages based on the profile.
 4. A method for creating abusiness simulation as recited in claim 1, where the individualizedcoaching messages include links to sites on the network.
 5. A method forcreating a business simulation as recited in claim 1, further comprisingthe step of providing the student with reference material in order tofacilitate remediation.
 6. A method for creating a business simulationas recited in claim 1, further comprising the step of selecting betweena plurality of topics based on the answers, wherein the topics areselected if the answers are incorrect and the selected topics arepresented in the individualized coaching messages to facilitateremediation.
 7. A method for creating a business simulation as recitedin claim 6, wherein the selected topics each have an associated segmentincluding at least one of text, audio and video clips, wherein thesegments of the selected topics are combined for being presented.
 8. Amethod for creating a business simulation as recited in claim 7, whereinthe segments include variables that are replaced with errors in theanswers.
 9. A method for creating a business simulation as recited inclaim 1, further comprising the step of highlighting error in theanswers.
 10. The method of claim 1, wherein step (b) comprises the stepof: (i) performing rule-based pattern matching.
 11. The method of claim1, wherein step (b) comprises the steps of: (i) asking at least onequestion about a current domain state; and (ii) determining a domainprofile in response to at least one answer that corresponds to the atleast one question, wherein the dynamic, goal-based remediation learninginformation feedback is in accordance with the domain profile.
 12. Anapparatus that creates a business simulation providing a goal basededucational environment, comprising: (a) logic for presentinginformation that motivates accomplishment of a goal for use in thebusiness simulation; (b) logic for monitoring, by a rule based expertsystem, answers to questions posed to evaluate progress of a studenttoward the goal; and (c) logic for providing dynamic, goal-basedremediation learning information feedback using a remediation objectcomponent, the remediation object component being a component of therule based expert system, the remediation object component including aknowledge base which generates individualized coaching messages based onthe answers that further motivate accomplishment of the goal.
 13. Theapparatus of claim 12, wherein the logic for monitoring comprises: (i) alogic component that performs rule-based pattern matching.
 14. Theapparatus of claim 12, wherein the logic for monitoring comprises: (i) afirst logic component that asks at least one question about a currentdomain state; and (ii) a second logic component that determines a domainprofile in response to at least one answer that corresponds to the atleast one question, wherein the dynamic, goal-based remediation learninginformation feedback is in accordance with the domain profile.
 15. Acomputer program embodied on a computer readable medium that creates abusiness stimulation providing a goal based educational environment,comprising: (a) a code segment presenting information that motivatesaccomplishment of a goal for use in the business simulation; (b) a codesegment monitoring, by a rule based expert system, answers to questionsposed to evaluate progress of a student toward the goal; and (c) a codesegment providing dynamic, goal-based remediation information using aremediation object component, the remediation object component being acomponent of the rule based expert system, the remediation objectcomponent including a knowledge base which generates individualizedcoaching messages based on the answers that further motivateaccomplishment of the goal.
 16. A computer program embodied on acomputer-readable medium that creates a business simulation as recitedin claim 15, wherein the remediation object component further includes aknowledge system and a software tutor.
 17. A computer program embodiedon a computer-readable medium that creates a business simulation asrecited in claim 15, and further comprising a code segment fordetermining a profile of the student, wherein the artificialintelligence engine generates the individualized coaching messages basedon the profile.
 18. A computer program embodied on a computer-readablemedium that creates a business simulation as recited in claim 15,wherein the individualized coaching messages include links to sites onthe network.
 19. A computer program embodied on a computer-readablemedium that creates a business simulation as recited in claim 15, andfurther comprising a code segment for providing the student withreference material in order to facilitate remediation.
 20. A computerprogram embodied on a computer-readable medium that creates a businesssimulation as recited in claim 15, and further comprising a code segmentfor selecting between a plurality of topics based on the answers,wherein the topics are selected if the answers are incorrect and theselected topics are presented in the individualized coaching messages tofacilitate remediation.
 21. A computer program embodied on acomputer-readable medium that creates a business simulation as recitedin claim 20, wherein the selected topics each having an associatedsegment including at least one of text, audio and video clips, whereinthe segments of the selected topics are combined for being presented.22. A computer program embodied on a computer-readable medium thatcreates a business simulation as recited in claim 21, wherein thesegments include variables that are replaced with errors in the answers.23. A computer program embodied on a computer-readable medium thatcreates a business simulation as recited in claim 15, further comprisinga code segment for highlighting errors in the answers.
 24. The computerprogram of claim 15, wherein the code segment that monitors comprises:(i) a set of computer instructions that performs rule-based patternmatching.
 25. The computer program of claim 15, wherein the code segmentthat monitors comprises: (i) a set of computer instructions that asks atleast one question about a current domain state; and (ii) a set ofcomputer instructions that determines a domain profile in response to atleast one answer that corresponds to the at least one question, whereinthe dynamic, goal-based remediation learning information feedback is inaccordance with the domain profile.